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ABSTRACT 

A study examined the effectiveness of five mandatory 
welfare employment programs in working with different segments of the 
Aid to Families with Dependent Children (AFDC) caseload. The study 
considered programs in San Diego, California; Baltimore, Maryland; 
several counties in Virginia; Little Rock and another county in 
Arkansas; and Cook County (including Chicago) in Illinois. The 
experimental groups consisted of persons who could receive the 
special service programs. Welfare recipients assigned to the control 
groups could not receive the services under study. They could, 
however, avail themselves of other community services (on their own 
initiative). The impacts and outcomes identified indicate that, if 
maximizing program impacts while limiting resources is the goal, it 
is a mistake to concentrate only on serving the most job-ready 
portion of the AFDC population. It also appeared relatively clear 
that programs should not focus exclusively on the most disadvantaged 
among the Work Incentive (WIN)-mandatory caseload (at lea^t not with 
low- to moderate-cost services). The data collected suggest that 
program operators wishing to maximize impacts on earnings may find it 
desirable to work first with applicant returnees or applicants with 
weak work records, whereas those operators desiring to maximize 
welfare savings devote increased effort to the most dependent client 
groups. (MN) 
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PREFACE 



This is a special report generated by research from MDRC s Demon- 
stration of State Work/Welfare Initiatives. This demonstration is a unique 
opportunity for MDRC to work with states in evaluating their mployment 
programs and thus to examine the potential effectiveness of a major 
component of recent welfare reform proposals. 

Using data from five state welfare employment programs (those in San 
Diego, Baltimore, Virginia, Arkansas, and Cook County, Illinois), the study 
presented here has two purposes, both of which are important in designing 
and operating effective programs. One is to produce reliable estimates of 
the programs 1 relative impacts on the employment and welfare receipt of 
different groups of welfare applicants and recipients. The second objec- 
tive is to help develop valid operational indicators for measuring the 
success of different welfare employment programs. 

The search for reliable and workable standards of performance to be 
used in employment programs for welfare recipients is one of the major 
themes in current efforts at welfare reform. MDRC hopes that these 
findings will contribute to informed decision-making on this subject and 
ultimately to the development and operation of even more eff ecitve programs 
designed to increase the self-sufficiency of all welfare recipients. 

Judith M. Gueron 
President 



EXECUTIVE SUMMARY 



This report presents an analysis of the effectiveness of five manda- 
tory welfare employment programs in working with different segments of the 
Aid to Families with Dependent Children (AFDC) caseload. Among their 
several goals, these programs all sought to increase earnings and decrease 
dependence on welfare, although local planners assigned different relative 
importance to one or the other objective. 

The analysis here has two purposes, both of which are useful in 
designing and operating programs. One is to produce estimates of the 
programs' relative impacts on the employment and welfare receipt of 
different groups of welfare applicants and recipients. These estimates may 
provide useful information to guide the targeting choices of policymakers 
who wish to maximize program impacts with limited budgets. The other 
objective is to explore the validity of certain principles of performance 
measurement in an effort to assist in the development of operational 
indicators that will best encourage the long-term objectives of maximizing 
earnings gains and reductions in welfare dependency. 

The Programs Evaluated 

The analysis is based on data collected in evaluations of welfare 
employment programs in San Diego, Baltimore, several counties in Virginia, 
Little Rock and one other county in Arkansas, and Cook County (containing 
the City of Chicago) in Illinois. These programs required the participa- 
tion of portions of the AFDC caseload which are "mandatory" under federal 
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Work Incentive (WIN) Program regulations (i.e., primarily women whose 
youngest child is six years old or older). The programs provided different 
services and operated in different labor markets, but all relied primarily 
on a combination of job search and work experience. Program costs (in 1987 
dollars) ranged frcro a low of $150 per experimental in Cook County to a 
high of $1,050 in Baltimore. A?DC income eligibility regulations also 
varied, with the highest benefit standards in California and the lowest in 
Arkansas. 

San Diego enrolled all WIN-mandatory welfare applicants but did not 
enroll persons who were already AFDC recipients. Participants went through 
a three-week job search workshop, followed by a 13-week work experience 
obligation for those who had not found an unsubsidized job. Baltimore 
enrolled WIN-mandatory applicants and persons who were recipients but had 
just become mandatory, usually because their youngest child had turned six 
years of age. Program activities could be selected from a number of job 
search, work experience, education and training options. The Baltimore 
program restricted active enrollment to 1,000 registrants per year during 
the period studied. 

Virginia enrolled its entire WIN-mandatory caseload. Job search was 
required of all enrollees and was followed, at county option, by short-term 
work experience, education or training. Arkansas enrolled its entire 
WIN-mandatory caseload, but only applicants and recipients who became 
mandatory after the research began were included in the impact sample. The 
program consisted primarily of independent and group job search and, less 
frequently, a work experience component. Arkansas also obtained a waiver 
enabling it to classify as WIN-mandatory AFDC mothers whose youngest child 
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was three to five years of age. 

Cook County, with one of the nation's largest urban caseloads, worked 
with recipients and the subset of applicants whose grants had been 
approved. As with Arkansas and Baltimore, the Cook County impact sample 
was restricted to those who became WIN-mandatory after the research began* 
The Cook County program included a two-month job search reporting component 
that relied mostly on the initiative of clients backed up by routine 
sanctioning for noncompliance. A private nonprofit work experience 
position was assigned at a later date for many of the individuals who did 
not find employment through the job search component. Work experience in 
all these programs typically lasted not more than three months; in no case 
was it designed to continue for as long as participants remained on AFDC. 

The Research Design 

All five evaluations used research designs in which eligible appli- 
cants and recipients were randomly assigned to experimental groups, which 
could receive the special program services, or to control groups, which 
could not. The experience of the control group members — who, on their 
own initiative, were able to avail themselves of services elsewhere in the 
ca unity — indicates what would have happened to the experimental groups 
in the absence of the special intervention, affording a benchmark for 
measuring program impacts. 

It should be noted that this study labels as an applicant any person 
who war- in the process of applying for welfare at the time of random 
assignment. Applicants retained the applicant designation even if they 
were approved and began receiving welfare; even if they left welfare; and 



even if their application was never approved. A recipient is anyone who 
was already receiving AFDC at the time of randan assignment. Recipients 
retained the recipient label even if they left welfare. 

The data on which the analysis is based were collected from 
Unemployment Insurance earnings records and automated AFDC payment ledgers 
for varying follow-up periods: a minimum of three years in Arkansas, two 
and a half in Baltimore and Virginia, and a year and a half in San Diego 
and Cook County. Average quarterly impacts are estimated on the basis of 
data from the fourth quarter after enrollment through the end of the 
follow-up period. The subgroup analysis focused on heads of single-parent 
households (primarily women). Two-parent households (mostly headed by men 
eligible under the AFDC-Un employed Parent program) were included in two of 
the individual program evaluations but are not included in this study. 

Impact estimates are reported on a per-experimental basis, even though 
only about half the exper imentals on average actually participated in seme 
formal activity. The estimates therefore represent the program effect 
averaged over all program registrants, not just participants. The esti- 
mates of average earnings and average AFDC payments also include all sample 
members, counting as zeroes those who did not work or did not receive 
welfare. Some special statistical considerations relevant to comparisons 
of subgroups are laid out in an appendix to the report. 

The analysis first examines impacts across subgroups that vary in 
their prior employment, welfare history and other demographic character- 
istics. It then uses subgroup impacts to evaluate two frequently used 
performance measures — the number of "job entries" (placements) and the 
number of cases "off-welfare" (case closures). 
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The Distinction Between Outcomes and Impacts 

The distinction between the meanings of ^outcomes" an3 "Impacts" as 
defined for this analysis Is critical to understanding the findings. An 
outcome Is the employment or welfare status of a person at a specified 
point after program enrollment. An impact is the change in an outcome 
produced by a program during that period. Program impact is estimated as 
the difference in outcomes between the experimental and control groups. 

Program imi ts are smaller than outcomes because the normal 
job-finding and welfare-departure rates of the AFDC population — i.e., 
control group outcomes — are not zero in the absence of a new program. 
But the relative difference between outcomes and impacts is not the same 
across all subgroups. Some subgroups exhibit worse-than-average outcomes 
but generate better-than-average impacts; other subgroups do the reverse. 

For example, in San Diego, experimental applicants with $3, COO or more 
earnings in the year before enrollment attained an average quarterly 
employment rate of 61 percent during the second year of follow-up. This is 
a high rate compared to other subgroups in these samples of AFDC mothers. 
But controls with the same prior earnings did almost as well, even without 
the special intervention. They averaged a 51) percent employment rate over 
the same follow-up period. The increase — i.e., the impact of the program 
— for these individuals was therefore only 2 percentage points. Clearly, 
the high outcome levels of employment reported for this group grossly 
overstate the influence of the program. In contrast, experimental 
applicants who had no prior earnings attained only a 30 percent employment 
rate, less than half that of the "more employable" group. This outcome 
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level, however, amounted to nearly an 8 percentage point increase over 
comparable controls. Thus, although the employment outcomes for this group 
seen on the face of it to be worse, their impacts are, in fact, larger. 
Analogous examples could be given for welfare outcomes* 

If the example discussed is not unique — and the research reported in 
the next section indicates that it is not — then policymakers are faced 
with a serious dilemma. On the one hand, they may deem it important to 
impress upon local operators that the ultimate program goals are employment 
and departure from welfare. On the other hand, by encouraging programs to 
strive for high rates of "placement" or "job entry" and high rates of 
welfare case closure, they may be driving operators to focus attention on 
groups of clients for whom impacts are below average. Thus, standards of 
performance based on simple outcomes, at best, may be unrelated to real 
program performance and, at worst, may even tend to undermine true 
effectiveness. 

The Major Subgroups 

Samples for the five programs were divided x. * o a variety of sub- 
groups, with impacts on earnings and welfare receipt estimated for each. 
The main objective of the analysis was to focus on subgroup definitions 
that might be of practical use in targeting program services. These 
subgroup definitions had to meet four criteria: (1) special targeting to 
the subgroup would not automatically be ruled out on political grounds; (2) 
the required background information is objective and verifiable; (3) the 
required background information can be obtained cheaply at program enroll- 
ment; and (4) the subgroup constitutes a meaningful share of the eligible 
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caseload. Prior earnings and welfare history turned out to be the subgroup 
dimensions that met these criteria and best predicted future employment and 
welfare receipt — the two outcomes of greatest interest for welfare 
employment programs. Several other individual characteristics were also 
investigated, including some which do not meet the four criteria for 
practical application in targeting but which are of interest nonetheless. 

The principal subgroup division used was by applicant/recipient 
status, with applicants further divided into first-time applicants and 
applicant returnees (i.e., applicants who had prior welfare experience, 
have gone off welfare, and have returned to welfare for sane reason). 
Within the applicant and recipient categories, three major subgroups were 
defined based on earnings from employment in the year prior to random 
assignment; no earnings, earnings of $1 to $2,999, and earnings of $3,000 
or more. Three other major subgroups were created according to length of 
time prior to randan assignment that the sample member had had her own AFDC 
case: never, two years or less, and more than two years. 

Obviously these characteristics can be combined in different ways to 
produce a variety of subgroup configurations. One particularly promising 
configuration is shown in Tables 1 and 2, grouped in three mutually 
exclusive tiers and arranged in roughly ascending order of welfare 
dependence and descending order of employability. The least dependent 
group, for example, is new applicants with no prior AFDC. The subgroups 
within a tier overlap, constituting alternative ways of grouping indi- 
viduals. Depending on location, the first tier comprised from 25 to over 
55 percent of the applicant sample, the second tier between 45 and 75 
percent. For the four programs that enrolled recipients, the entire third 
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TABLE 1 



SUMMARY OF IMPACTS OH QUARTERLY EARN' 1 NGS FOR MAJOR 
SUBGROUPS OF AF DC APPLICANTS AND RECIPIENTS 

1 



Subgroup 


Son Diego 


Bo 1 t ircore 


V i rg i n i o 


Arkonsas 


Cook 

o 

County 


Fi rsl Tier 












Appl i conls wi th No 
Prior afd r 


< j. 0 7 
* + J / 


S + 1 L \ 


$ - 1 3 


$ +26 


5 


Second Tier 












Appl i conl Returnees 


+158** 


+188*** 


+ 1 14* 


+211*** 





Appliconl Returnees with 
Less thon $3000 Pr i or 
Eo rn i ngs 


+ 15 1 * 


+ 7 c o * * * 


T L U 






Th i rd Tier 












All Recipients 




+ 3 7 


' O 7 


4- 1 0 
T 1 7 




ReciDientS with More than 

■vw^vkrlvlflv ill III Hi V § w | || U II 

Two Yeors on AFDC 





-0 


+110** 


+ 14 





Rec i p ien ts with No 
Prior Earnings 





+ 1 04** 


+ 70 


+ 29 


+ 12 


Rec ipients wi th No Prior 
Eornings ond More thon 
Two Yeors on AFDC 




+ 88 


+ 94* 


+ 28 




Al 1 AFDC 












Quorterly Eornings impoct 


+ 1 18** 


+96*** 


+ 72** 


+ 70** 


+ 19 


Averoge Con* ol -Group 
Eo rn i ngs 


773 


634 


541 


257 


451 



NOTES: Tiers ore mutuoiiy exclusive; subgroups within tiers overlop. All 
vo I ues ore overoges for the fourth through the lost quarter of follow-up. 
Estimotes include zero volues for somple members not employed or for somple 
members not receiving welfore. 



A two-toiled t-test wos appl i e d to differences between experimental 
ond control groups. Stotisticoi significance levels ore indicoted os: * = 10 
percent; ** = 5 percent, *** = 1 percent. 

°The definitions of 'opplicont' ond 'recipient' for Cook County ore 
not strictly comporoble to those of the other progroms. 



TABLE 2 



SUMMARY OF IMPACTS ON QUARTERLY AFDC PAYMENTS FOR MAJOR 
SUBGROUPS OF AFDC APPLICANTS AND RECIPIENTS 



Subgroup 


Son Diego 


Bo 1 t imor e 


Vi rginio 


Arkansas 


Cook 
County 


Fi rsl Tier 












Appl i conts with No 
Prior AFDC 


$ -5 


$ -9 


$ -28 


$ -31 


$ „- 


Second Tier 












Appl icont Returnees 


-47 


-15 


-16 


-19 





Applicont Returnees with 
Less thon $3000 Prior 
Eo rn i ngs 


-63* 


-19 


-29 


-22 


• 


Th i ro Tier 












Al 1 Recipients 




+ 5 


-24 


-60*** 


-13 


Recipients Kith More than 
Two Yeors on AFDC 




+ 19 


-48** 


-44* 


— — — 


Rec i p ients with No 
Prior Earnings 





+ 1 


-26 


-63*** 


-6 


Por i r ion t C with 1'n D r ■ n r 
KvL 1 p 1 ell i i Willi IS U rl 1 or 

Eornings ond More thon 
Two Yeors or, AFDC 




- 1 


-48** 


-48* 




Al 1 AFDC 












Quorterly AFDC Poyments Impoct 


-33 


-5 


-23* 


-40*** 


-13 


Averoge Cont ro! -6 roup AFDC 
Poyments 


469 


501 


345 


232 


646 



NOTES: Tiers ore mutuolly exclusive; subgroups within tiers overlop. All 
volues ore overoges for the fourth through the lost quarter of follow-up* 
Estlmotes include zero volues for somple members not employed or for sample 
members not receiving weifore. 

A two-toiled t-test wos opplied to differences between experimental 
ond control groups. Stotisticol significance levels ore indicated as: * = 10 
percent; ** - 5 percent, *** = 1 percent. 

°The definitions of 'applicant* ond 'recipient' for Cook County ore 
not strictly comporoble to those of the other progroms. 
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tier accounted for about 40 to over 65 percent of the full sample. 

Some idea of the relative eraployability and welfare dependence of the 

different tiers is given by the experience of control group members — 

specif ically, their employment in quarters 4 to 6 after enrollment and 

th*ir welfare receipt in quarter 6. For the first tier group, for example, 

37 percent of controls were employed and 31 percent were receiving welfare. 

■ 

For the second tier groups, 28 to 37 percent of the control groups were 
employed and 46 to 49 percent were receiving welfare. For the third tier 
subgroups, 13 to 23 percent were employed, and 72 to 80 percent were on 
welfare. 

In assessing the findings, attention should be paid to the magnitude 
of impacts as well as their statistical significance. One way to do this 
is to compare a subgroup impact to the approximate median value of all 
subgroup impacts in the five samples, i.e., the value for which about half 
the impact estimates fall above and half below. Statistically significant 
impacts above this overall average (roughly $100 per quarter for earnings 
increases and $20 per quarter for welfare savings) are called "above 
average." Impacts below this level are called "below average." 

In interpreting the results, it should be born in mind that (a) the 
samples consist primarily of adult women without pre-school children and 
(b) the interventions observed were mandatory, mass participation programs, 
with a low-to-moderate cost per enrollee. The findings may not be directly 
general izable to other poverty groups or to other kinds of programs. For 
example, information abcut a client's past earnings and AFDC receipt may 
have less significance for youth, who typically have only short work and 
welfare histories. 
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Impacts for Subgroups Defined by Prior Employment and Welfare History 

• The groups that were most job-ready and least welfare-depend- 
ent, as defined by previous work and welfare experience, had 
below-average program impacts that were generally not 
statistically significant. 

Impacts for the first tier subgroups were generally low. Applicants 
with no prior AFDC history — first-time applicants — had quarterly 
earnings gains over $100 per quarter in only one sample, and these were not 
statistically significant. (See Table 1.) Welfare savings were also small 
for this group and were never statistically significant. (See Table 2.) 

Another subgroup (not presented in Tables 1 anc 2) which might be 
considered in the first tier is applicants with $3,000 or more in prior- 
year earnings. This group showed significant earnings gains in only one 
sample and never showed welfare savings. Additional analysis of the one 
location with earnings gains revealed that nearly all of the increase 
accrued to individuals who would have left welfare even without the special 
intervention. This suggests that, even when the least dependent do achieve 
earnings gains, these may not translate into reductions in AFDC receipt. 

• Earnings impacts were found most consistently for individuals 
in the mid-dependency tier. 

The second tier contains applicants with seme prior welfare history 

(i.e., returnees). Also shown separately in Tables 1 and 2 is the subset 

of these returnees who earned less than $3,000 in the year prior to their 

application for welfare. Compared to the other two tiers, these groups 

proved more likely to have above average and stat ically significant 

earnings impacts, although this was not true in all cases. Moreover, 

additional statistical tests showed that earnings impacts f >r applicant 

xvii- 1 S 




returnees were significantly greater than those for new applicants and 
recipients combined in two out of the three cases where such comparisons 
were possible. 

• Earnings impacts were not found consistently for subgroups in 
the most dependent tier. 

The third tier contains the full sample of welfare recipients in each 
of the four locations where they were enrolled, with results also presented 
for several overlapping subgroups of recipients. Employment and earnings 
impacts were found for some of these subgroups, including some gains that 
were statistically significant. However, these were not consistent and 
were rarely above average. This was not a result, however, of the 
programs 1 failure to serve this population. On the contrary, participation 
rates for these groups were as high as, or higher than, the rates for 
applicants. 

• Statistically significant welfare savings were found for 
several of the more dependent subgroups, but this was not 
consistent across samples. Welfare savings were also found 
for some of the mid-dependency groups, although generally 
these were not statistically significant. 

Welfare savings, because they were smaller than earnings increases, 

were more difficult to contrast across subgroups (see Table 2). Modest 

savings were found for subgroups in the second tier. The individual 

estimates were usually not statistically significant. Interestingly, in 

two samples (Virginia and Arkansas), some of the more dependent subgroups 

showed welfare reductions that were relatively large, especially in 

comparison to their earnings gains. 

• The findings do not support a strong recommendation to narrow- 
ly target low-to-moderate cost services. Where increasing 
earnings is the primary program objective, subgroups in the 
mid-dependency tier may be the best candidates for priority 
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attention* Where program objectives emphasize reductions in 
welfare payments, groups in the most dependent tier may assume 
increased importance. 

No subgroup emerges clearly and consistently as appropriate for 
exclusive targeting of lcw-to-raoderate cost services. While not 
conclusive, the findings suggest, moreover, that selection of a targeting 
strategy might depend on the primacy of different potential program 
objectives. When resource? are scarce, administrators seeking to increase 
the earnings of those on welfare might assign priority for services to 
subgroups in the second^ tier. These groups — applicant returnees — 
showed the most consistent earnings gains. Alternatively, administrators 
seeking welfare savings should recognize that a substantial share of 
estimated savings were found in the high-dependency third tier. 

The particular programs tested, however, suggest caution in using 
these findings to reach a final conclusion on targeting. The results cane 
from low- to-moderate cost programs that explicitly sought to include the 
full range of individuals within the groups served. Inclusion of the more 
job-ready, tier-one registrants might not be suggested by the impact data, 
yet this group may have played an important role by providing encouragement 
to the ±ess job-ready participants and to program staff. Further, adminis- 
trators have a choice not only in who they target but in the services they 
provide. For recipients in tier three, lcw-to-moderate cost programs do 
not consistently produce above-average earnings gains. It remains unclear, 
however, whether more intensive or different services could lead to more 
consistent earnings impacts, while also producing welfare savings. 
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Other Subgroup impacts 

In addition to prior work and welfare history, subgroups were defined 
along several demographic dimensions, including marital status, education, 
and the number and ages of children. These factors, although important, 
were less strongly related to future employment and welfare receipt. 

• The samples yielded often conflicting impact estimates for 
subgroups of the WIN-mandatory AFDC caseload defined on 
characteristics other than prior employment and welfare 
experience. 

High school diploma status, absence of children under age twelve ; 
number of children, age and ethnicity did not produce consistent impact 
differentials across program samples. Among sample members without a 
diploma, earnings impacts were found only for applicants and then only in 
Baltimore and Virginia. The education and training services available in 
these two programs may have played a role in achieving results for dropouts 
there, but this is by no means a clear-cut conclusion, particularly since 
controls in Virginia were able to obtain similar services on their own at 
about the same rate as experimentals. 

There was sane evidence that the bulk of welfare savings were obtained 
for women who were not married at the date of random assignment, particular- 
ly those who were never married. Evidence also suggests that impacts among 
recipients in rural areas were weak, and that this may account for some 
part of the differences between applicants and recipients overall. 

• In Arkansas, mandatory status was extended to women whose 
youngest child was three to five years old. Impacts for these 
women were about the same as impacts for the regular 
WIN-mandatory enrollees. 

The inclusion of women with children ages three to five in Arkansas 

more than doubled the number of individuals who enrolled in the program 
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daring the demonstration. Employment rates were the same for this group as 

for women with older children. Program impacts on earnings and welfare 

receipt were also similar. The total effects of the program on the AFDC 

caseload were therefore more than twice what they would have been if only 

the impacts on regular WIN mandatories were counted. Scrae caution should 

be exercised in generalizing this finding to other program contexts, 

however, since AFDC grant levels are low ir. Arkansas and the sample size 

was relatively small. 

• The combination of impact differences across dependency sub- 
groups and according to demographic characteristics suggests a 
possible threshold effect for earnings impacts. 

For cases whose multiple disadvantages combine to make them more de- 
pendent than sane threshold level, the typically low-cost services provided 
by the programs in this study may bejin to lose their effectiveness in 
increasing earnings. It is not clear how large the group below the thres- 
hold may be, or whether it can be adequately identified with demographic 
data alone. It was found, however, th£t sample members who were recipients 
with more than two years on welfare, ithout recc c earnings, and with no 
high school diploma attained below-average earnings impacts in the three 
samples for which data on these characteristics were available. 

Program Performance Measures 

'In this study, as noted, a program's performance is defined as the 
impact it achieves. Normally, administrators do not have impact measures 
available to them. Instead, measures of outcomes or participation must be 
utilized to set operational goals and standards of performance. The most 
popular of these performance indicators have been "job entries* (place- 



merits) and cases "off welfare - (case closures). For programs that seek to 
maximize impacts, these performance indicators can only be effective if 
they are related to impacts. 

The validity of performance measures based on observable employment 
and welfare outcomes was assessed by examining the correlation between job 
entries and case closures and program impacts. Unemployment Insurance 
earnings were used to identify a job entry; AFDC payments were used to 
determine whether a sample member went off welfare. 

• Unadjusted job-entry and off -welfare measures are not 
empirically valid indicators of performance if program 
objectives are to increase earnings and decrease welfare 
receipt. 

No consistent relationship emerged across program samples between 
simple outcome indicators (i.e., job entries and esse closures) and program 
impacts. in San Diego and Baltimore, in fact, subgroups with the higher 
job-entry scores had lower earnings and welfare impacts. in Virginia and 
Arkansas, job entries indicated employment impacts mostly for sample 
members with relatively lew risk of remaining on welfare a long time. Job 
entries were therefore negatively related to in^acts on welfare receipt. 

The study identified two raa^or reasons why outcome measures are not 
likely to be good indicators of program impacts. First, outcome measures 
substantially overstate true impacts since many program registrants would 
have found jobs or left welfare on their own. Second, the overstatement 
was not uniform across subgroups. Moreover, outcome differences among 
programs were determined more by characteristics of the enrollees, local 
AFDC eligibility regulations and local labor market conditions than by the 
size of program impacts. 
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• The use of longer-term follow-up information about employment 
and welfare receipt did not improve the correlation of either 
the job-entry or off-welfare measures with impacts in aost 
cases* 

One possible strategy for improving performance indicators is to 
increase the length of follow-up data over which the indicators are calcu- 
lated. Tests of indicators based on longer-term data (up to three years 
after random assignment) were not successful in this respect. Moreover, 
the "short-terra" outcome measures that were tested made use of follow-up 
which was already longer than that available to most program operators, who 
often have only the enrollee's status at date of termination from the 
program. 

• Weighting outcome measures by prior work and welfare history 
improved the relationship between performance indicators and 
impacts in some cases. Still, the development of weighting 
schemes valid for a variety of program models and local 
conditions should go beyond this preliminary research. 

Giving more weight to job entries and movement off welfare for individ- 
uals with weaker recent work experience improved the correlation between 
outcome measures and impacts on earnings and welfare receipt. Weighting 
for longer prior welfare experience also yielded some improvement. Weight- 
ing — whether with simple weights or complex regression-adjustment 
formulas — tends to correct the adverse allocation properties of outcome 
indicators by increasing the incentives to work with less job-ready, more 
dependent eligibles. At the same time it retains the focus on employment 
and welfare receipt. 

These results should be considered only as one test of the general 
principal of weighting. The particular weighting schemes tested should not 
be viewed as the best of all possible schemes. Much remains to be learned 
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about client behavior before a definitive set of variables and weights can 
be confidently accepted. Moreover, the variety of program approaches and 
local welfare and labor market conditions may mean that different perform- 
ance measurement systems will be better in sane circumstances than Jn 
others. 

• Simple participation measures can also give misleading signals 
to program operators. Monitoring participation separately for 
the major subgroups or adopting weighted participation 
measures may prove more efficient. 

Performance measures based on participation — that is, activity in 
program services — are sometimes proposed as an alternative to job-entry 
and off-welfare outcome measures. In recent years, participation standards 
have been criticized as being less directly related to the program goals of 
employment and welfare departure. Nevertheless, participation measures do 
have some administrative advantages as indicators of performance. Partici- 
pation can be readily observed and immediately reported, assisting manage- 
ment in monitoring day-to-day operations. For mandatory programs, monitor- 
ing participation may be deemed useful in ensuring consistent treatment of 
clients across subgroups and local offices. For a program with a potential- 
ly large base of eligibles, monitoring participation separately by subgroup 
can at least ensure that groups with documented impacts are being reached. 

Findings of this study suggest that a distinction should b«2 drawn 
between "maximtm participation" and "efficient participation." The sub- 
group results imply that efforts to maximize total participation may be 
less efficient than efforts directed towards increasing participation among 
the moderately dependent (i.e., second tier) subgroups. Extending partici- 
pation downward into second-tier and third-tier subgroups should increase 



program effectiveness more than extending participation upward into the 
first tier. Monitoring participation separately for the major subgroups or 
weighting participation in the same fashion as outcomes should help achieve 
that goal. 

Conclusions anu Unresolved Issues 

The research reported here addresses a number of important issues in 
the monitoring and targeting of welfare employment programs. It also 
raises questions relevant to the broader employment and training delivery 
system. TLj results are suggestive rather than definitive. In soik cases, 
the implications are quite clear. But in others, they raise questions to 
which the appropriate policy response is less certain. 

Subgroup impacts . The findings are clear that, if resources arc- 
limited and maximizing program impacts is the goal, it vis a mistake to 
concentrate only on serving the most job-ready portion of the AFDC case- 
load. Since thiF s * ^ tendency in the V7IN program, this message is an 
important one a warrants a shift in strategy. The evidence favors the 
establishment of program goals that encourage working with more dependent 
and less job-ready individuals. Many administrators are already recog- 
nizing this lesson and are adjusting service priorities accordingly. 

It is also relatively clear that programs should nut focus exclusively 
on the most disadvantaged among Lhe WIN-mandatory caseload, at least not 
with .low- to-moderate cost services. The results do not provide conclusive 
guidance to program operators if resource constraints require then to 
choose* among groups of eli^ioles. There is sane evidence that the 
selection of a target \ \pproajh might depend on the importance attached 
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to different potential program objectives. 

Operators who wish to maximize impacts on earnings may find it 
desirable to work first with applicant returnees or applicants with weak 
work records. With additional resource* they might next expand services 
to include longer-term recipients. Operators who seek to maximize welfare 
savings may want tc devote increased effort to the most dependent groups. 

The nature of the state initiatives tested, however, suggests a nunber 
of cautions in using the results of this study to reach a conclusion on 
narrow targeting. First, the results reported in this analysis were for 
programs that did not target narrowly but rather served individuals with a 
wide range of prior work experience and other factors affecting eraploya- 
bility. It is possible — particularly in group job search components — 
that the presence of at least seme job-ready enrollees encouraged both 
program staff and the more disadvantaged, thereby contributing to the 
positive results reported here. 

This "ma ^streaming" hypothesis is not tested in this study, but it 
suggests that administrators should look carefully at the operational 
results of more targeted services before using resources exclusively for 
individuals in the second cr third tiers. Working only with individuals 
with lower skills and measured outcanes could have political, administra- 
tive or stigmatizing effects. For example, it may be difficult to convince 
people that a placement rate of 30 percent could represent a substantial 
positive achievement for more disadvantaged groups, even though this is 
suggested by the impact findings. Such lew rates may also discourage staff 
efforts. Finally, employers may think differently about a program that 
refers only clients with no prior work history. 
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A second reason for caution is that, in allocating resources, program 
operators have a choice not only in who is served but also in what services 
are offered. The results reported here suggest that there may be a 
threshold effect on the earnings (although not the welfare) impacts of 
low-to-raoderate cost programs: They may have comparatively limited impact 
on the earnings of the most dependent groups. In the future, it will be 
important to examine whether more intensive services can lead to larger 
earnings impacts for these individuals. Results from the National 
Supported Work Demonstration, for example, have shown that earnings 
increases can be obtained with more extensive services for certain groups 
with long welfare histories. But this was a small-scale, voluntary 
program. Its findings may not generalize to more broadly-based 
interventions. 

Performance indicators . This study supports the increasing recog- 
nition that alternatives to unadjusted outcome measures are needed to 
establish valid performance standards. Weighted measures appear to create 
more appropriate incentives for program operators by explicitly taking 
account of participants' individual differences. Preliminary evidence 
suggests that selecting the appropriate characteristics for weighting is at 
least as important as precision in calculating the weights assigned to 
those characteristics. When applied to adult welfare recipients, weighting 
schemes should, at a minimum, include prior employment or welfare history 
or both. 

Nevertheless, even weighted performance indicators have limitations. 
First, although weighting by demographic charactertisics may correct sane 
of the adverse properties of unweighted outcome indicators, it is not 
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likely to provide a perfect solution to the problem of targeting. For any 
given enrollee, demographics alone cannot predict with precision who will 
succeed by participating, and only by participating. The rough guidance 
provided by weighted indicators may be sufficient for allocating low-cost 
services; it may not be sufficient for allocating high-cost services. 
Better data on literacy, general and specific work skills and family 
circumstances might help identify those who can benefit from particular 
services, but the potential usefulness and cost of these data are not 
addressed in thi report. 

Second, the results presented here come from broad coverage programs 
charged with enrolling and working with everyone within a specified group 
of welfare recipients. Very different issues and lessons could arise in 
selective programs not intended to reach all persons who are categorically- 
eligible. These include programs where participation is voluntary on the 
part of the client or where local managers can choose the individuals they 
wish to enroll. In such cases, merely weighting performance measures may 
have limited effect: Program operators could, for example, screen intense- 
ly among the more disadvantaged, identifying only the most able and motivat- 
ed within the heavily weighted groups, thereby undercutting the objective 
of serving those who are really less likely to succeed without help. 

Third, it will be difficult to develop performance measures that allw 
meaningful comparisons of effectiveness between programs and across time. 
A range of factors, such as the local economy and AFDC benefit levels, can 
have a substantial effect on the composition of the caseload and job 
prospects for program participants. Comparing program ef f ectiven -«s with 
unadjusted measures can be very misleading; further empirical work is 
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needed to determine how reliable such comparisons can be when adjustments 
are made* 

Finally, performance measures are only useful if they can be imple- 
mented. The necessary data must be obtainable quickly and at reasonable 
post. The calculations must be straightforward enough to be accessible and 
useful to line staff. The analysis of welfare, and especially earnings, in 
this report drew on data bases that may only be available to program 
administrators with considerable lag, if at all. Cost of data collection 
and ease of interpretation are likely to be important factors in designing 
measures that are both feasible and valid indicators of true program 
performance. 
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CHAPTER 1 



INTRODUCTION 

The search for reliable and workable standards of performance to be 
used in employment programs for welfare recipients has been one of the 
major themes in current efforts to reform welfare policy. Such close 
attention is warranted because performance standards are one of the primary 
means by which broad policy is translated into the specific objectives that 
guide the operations of programs. Performance measures fulfill a monitor- 
ing function, allowing administrators to assess how well existing programs 
are doing and to identify problems in the delivery of services. They also 
fulfill an allocation or targeting function. By encouraging a focus on the 
welfare groups most likely to help the programs achieve a high performance 
rating, they influence programs 1 service priorities and thereby the alloca- 
tion of funds. To the extent that program objectives include maximizing 
impacts on individuals, the study of performance measures is bound up with 
the study of program impacts on subgroups. 

This report examines questions about subgroup effects and performance 
indicators by studying five employment and training programs for recipients 
of Aid to Families with Dependent Children ( AFDC) . In all these programs, 
participation was mandatory. This is the final report of a two-part 
investigation into the differences among the impacts of these programs on 
the employment and welfare receipt of selected AFDC subgroups. A previous 
report presented complete subgroup analyses for only two of the programs, 
plus a preliminary analysis for the third program. 1 
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The study uses data from the Demonstration of State Work/Welfare 

Initiatives, a five-year, eight-state series of large-scale social 

experiments conducted by the Manpower Demonstration Research Corporation 

(MDEC). The data are unusual in that the research samples they describe 

were generated in controlled experiments based on random assignment. The 

research focuses on program performance defined in terras of effects on 

employment and welfare receipt. In addition, an analysis of program costs 

2 

was undertaken for the principal subgroups in two of the programs. This 
is reported in Chapter 2, 

It should be emphasized that the subgroup study focuses only on AFDC 
single parents (mostly women) meeting the Work Incentive (WIN) program 
definition of mandatory: These were parents who had no child under the age 
of six (under three in Arkansas) and no other known barriers to partici- 
pation. This WIN-mandatory group makes up just over one- third of the AFDC 
caseload nation wide. Unemployed heads of two-parent households who are 
also WIN-mandatory were typically part of the MDRC research samples in the 
states that had an AFDC-U program and therefore served this group. These 
samples have been excluded from the subgroup study, however, because their 
behavior is typically too different from the AFDC group to be analyzable as 
part of that group. They receive assistance under different rules f ran 
those applying to the AFDC single-parent case heads, for example; they have 
different labor supply patterns; and, because WIN-raandator ies under AFDC-U 
are primarily married men, they have different work backgrounds. The 
Arkansas program extended mandatory status to AFDC case heads with a 
youngest child aged three to five. The results for this group are 
included, and are of particular interest in the context of the current 
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policy debate. 

The implications of the analysis are somewhat broader in scope than 
mandatory welfare employment programs because many of the issues examined 
are common to other programs for lw-incane or disadvantaged groups, such 
as those funded by the Job Training Partnership Act (JTPA). But some care 
should be taken in generalizing from the conclusions. In particular, it 
should be borne in mind that the programs studied were usually intended for 
mass coverage, and the cost per enrollee was relatively low. Performance 
standards and targeting strategies may differ for services that have higher 
per enrollee costs. 

In some respects, the role of targeting has less to do with the 
voluntary/mandatory distinction per se than with the extent to which the 
program is given discretion to select its participants. Programs expected 
to serve all or substantially all those who are technically eligible will 
have less opportunity for targeting than programs whose primary purpose is 
to maximize impacts on earnings or welfare receipt for only a portion of 
the eligible population, for example if resources are not sufficient to 
serve everyone. In the former case, the "requirement - nature of the 
program applies not only to the clients but also to the operators if they 
are required by law to enroll all eligible persons. 

For such broad coverage programs, selective targeting may be 
inconsistent with program objectives. The goal would not be to narrow the 
focus of a program, but rather to extend it to groups of eligib.les who have 
had lew participation rates in the past. For example, a mandatory "work- 
fare" program, whose goal is the payback of welfare benefits through 
community service by all able-bodied recipients, may monitor particular 



-3- 




33 



groups to ensure that inequities are not created through unequal 
application of the participation requirement. The program may also target 
a group for special attention to raise its participation rate if it is 
below average. 

In contrast, selective coverage programs working with a narrow sub- 
group of welfare recipients expected to reap the largest benefits fran the 
program may be mandatory, but are typically voluntary programs. Because 
selective or partial coverage programs are not intended to serve everyone, 
targeting may be the paramount design question, and performance standards 
that carry implicit incentives for targeting in a particular fashion may be 
one of the primary influences on how a program is implemented and what its 
impacts on participants are. In addition, if resources are too limited to 
serve everyone, targeting may help programs to use resources efficiently. 

This report first examines differences in impacts across subgroups 
that vary in their prior employment and welfare history, and in a variety 
of individual characteristics (education, age, number of children, and the 
like). The intent of the comparison is to identify whether the relative 
impacts on any subgroups are consistent across the five programs. To the 
extent that they are, more general conclusions can be drawn about which 
subgroups benefit the most from the types of programs included in the 
evaluation. 

The analysis then uses the subgroup impacts generated fran the experi- 
mental data to evaluate the validity of two frequently used performance 

3 

measures — the number of "job entries* (placements) and the number of 
cases 'off-welfare* (case closures) — and variants of them. The issues 
addressed in this report also apply to other measures of performance, such 
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as wage rates, job retenticn, and program participation. 

The discussion is structured as follows. The rest of this chapter 
reviews issues relevant to welfare population subgroups ana program 
performance. Chapter 2 discusses the welfare employment programs studied 
and their research designs. Chapter 3 explains the methodology used to 
estimate subgroup impacts and to test performance indicators. Chapter 4 
presents impacts for the major subgroups in the study. Chapter 5 evaluates 
the validity of several alternative formulations of employment- and 
welfare-rela^d performance measures, using program impact estimates. 

I. Issues in Program Performance 

Recent research on relatively 3cw-cost welfare employment programs has 
put issues regarding ptcaram performance into sharp focus. Earlier studies 
as part of the MDRC Work/Welfare Demonstration are illustrative. Local 
experiences vary widely , but typical employment rates for a group of pro- 
spective enrollees without the program mi^.it average -\bout 30 percent over 
a three-iuonth period. The programs cost in the range of $150 to $1,000 per 
experimental. After the programs were established, about half of those 
enrolled typically participated in some formal activity. The overall 
employment effect on the enrollee groups, counting nonparticipants as well 
as participants, averaged an increase in the employment rate of around 4 
percentage points, from a 30 to a 34 percent employment rare. 

Not all who are enrolled in a program benefit from it. Sane who find 
jobs would have found them without the program. Conversely, seme who 
participate do not benefit. Not all who are given the chance to 
participate do so. Seme leave welfare for reasons unrelated to program 
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participation or even to employment (such as getting married or moving). 
One of the possible goals of targeting, therefore, is to concentrate 
resources on the groups most likely to benefit fror the program, or most 
cost effective to serve. 
A. Targeting 

Much of the recent work in targeting welfare employment programs has 

focused on AFDC subgroups outside the WIN-raandatory category — such as 

4 

mothers with young children. It has been found that length of welfare 

dependency is related to objectively measured individual characteristics 

and that the majority of people who enter the welfare system spend less 

than four years on the rolls, even counting repeat spells. The minority of 

people who remain on welfare for many years account for the bulk of AFDC 

benefit expenditures; one study estimated that as much as 60 percent of all 

grant outlays are paid to only 25 percent of all recipients. 5 

Young mothers, particularly never-married mothers, appear to be at 

especially high risk of long-term dependency. The conditions and problems 

that lead to extended dependency, however, may not be amenable to change 

6 

with low-cost employability services. In any case, the study has only 
limited ability to address this most dependent group and the services that 
may be most effective for them for two reasons. First, they are typically 
not in the traditional WIN-mandatory category, the focus of the programs 
being compared here. "Most-dependent ■ as used in this report refers only 
to the most dependent of those among the WIN-mandatory portion of the AFDC 
caseload. Only one of the programs studied here includes substantial 
numbers of women with children under six years old. Second, the programs 
did not emphasize sane services (such as day care) that might be of 



particular value to working women with preschool children. 

The results of this study are most useful for planning programs 
similar to those evaluated here: relatively low-cost services designed for 
the WIN-mandatory portion of the AFDC caseload. In program planning, the 
cost per person for a particular kind of intervention will normally be a 
given. If this is combined with a fixed total budget, the number of 
persons who can be served is also determined by the following simple 
identity: 

NUMBER OF PERSONS SERVED = (TOTAL COST)/ (COST PER PERSON). 

In cases where the number that can be served is smaller than the total 
number eligible, information from a study such as this about relative 
program effectiveness across different subgroups may be useful in deciding 
who should be given priority for services. 

It should be noted that narrowing the target group within the WIN- 
mandatory population carries many uncertainties. Resources may be freed 
up, but restricting the target group may reduce the total effect of 
services. T'lis is because the impact of a program will automatically be 
zero for those who are excluded frcm it. The total effect of a service is 
given by the simple identity: 

TOTAL IMPACT = (IMPACT PER PERSON) X (NUMBER OF PERSONS). 

Targeting may very well increase the impact per person but, by definition, 
decreases the number of persons served. Targeting one-half of the 
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eligibles, for example, means that average impact per target group member 
must be twice the average impact for all persons taken together to achieve 
the same overall impact. 

There is even sane chance that narrow targeting — e.g., on the more 
disadvantaged — may reduce program effects on the target groups. This is 
equivalent to the question of "tracking" versus "raainstreaming, " so 
prominent in education. An open question in welfare employment programs, 
for example, is whether loosely structured, low-cost services, such as job 
search workshops, can be as effective if women with no prior work experi- 
ence do not have the opportunity to learn from others who have held jobs. 

B. Measures of Performance 

Performance measures are intended to promote program effectiveness, 
conserve resources, and ensure compliance with overall goals and direct- 
ives. *joth in monitoring and in the allocation of funds, performance 
standards play a critical role in determining the efficiency of program 
expenditures; and, in a period of fiscal restraint, it is particularly 
important to choose performance measures that increase rather than decrease 
efficiency. Poorly design^ or inadequately tested performance standards 
can work against the objectives of the authorizing legislation. They can 
promote methods of operations that waste staff time and other program 
resources, with the result that neither the welfare population nor society 
is well served. 

A wide range of indicators has been developed and used in the WIN 
program, programs funded by the Comprehensive Employment and Training Act 
(CETA) and, more recently, programs funded by JTPA. Historically, job 
placements and welfare reductions have been the most important indicators 
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in WIN. These measures have c^eraed useful in conveying program achieve- 
ments in straightforward terms to policymakers and the general public. 
Their incorporation into the fiscal WIN Allocation Formula underlined their 
significance to operators of welfare employment programs. Other indicators 

were also part of the WIN Allocation Formula, such as the quality of job 

7 

entries, usually measured by wage rates and job retention. 

These indicators all measure the outcomes of a registrant's program 

experience at scne point after registration. Another set of indicators 

looks at the activity of registrants while in the program; these include 

counts of registrants, participants, program completers and similar 

measures. Participation data ha~ T e been examined in evaluations of WIN, 

CETA and other programs. Th^ trend recently has been to deeraphasize these 

indicators, even though they provide immediate feedback and the required 

information is relatively inexpensive to collecc. Instead, emphasis has 

been on measures that communicate program goals in terms of post-program 

outcomes. For example, the JTPA legislation explicitly requires that 

standards for adult participants bo based on job entries, wages and 

9 

earnings, retention and welfare reductions. 
C. Outcomes and Impacts 

The distinction between outcomes and impacts is critical to an under- 
standing of program performance. An outcom e is the employment and/or 
welfare status of a person at some point in time after program registra- 
tion. Hence, the outcome "employed and not receiving welfare at quarter 4" 
describes the status of a person 9 to 12 months after program entry. The 
real effects of a program cannot be judged by outcomes, however, given the 
high degree of normal job-finding and welfare departure within the welfare 
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population (i.e., outcomes that are not related to program experience). 

Program impact is the effect of the program itself. It is the differ- 
ence between outcomes with the program and outcomes without it. This study 
estimates program impacts as the difference between the outcomes of a 
randomly selected group of people eligible for the program treatment (the 
experimental group) and the outcomes of a similar group of people not 
eligible for the treatment (the control group). The distinction between a 
level (the outcome) and a difference (the impact) is fundamental. 

Past research has suggested that groups exhibiting worse-than-average 
outcomes may, in fact, experience better-than-average program impacts* For 
example, an MDRC evaluation of a jjb search and work experience program 
operated in San Diego found that 73 percent of WIN-mandatory AFDC appli- 
cants who had worked at sane time during the year prior to their progra~ 
entry were able to find employment during the 18 months following enroll- 
ment. This outcome was, in fact, only a 2 percentage point increase (or 
impact) over the control group employment outcome — that is, the rate that 
applicants with a prior work history were able to achieve on their own. In 
contrast, of the enrollees without prior employment only 48 percent were 
able to find employment; but this outcome was a 10 percentage point 
increase over the control group's employment rate of 38 percent.*** 

Given these patterns, performance indicators based only on outcomes 
create a misleading impression of program effectiveness. They may also 
lead to ineffect'.*) targeting of program resources if these standards place 
emphas on serving the least appropriate groups — that is, those who 
would .h ve done well on their own without the program. Conversely, people 
who could benefit most from these programs may be underserved. 
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D, Evaluating Performance Indicators in Light of Impact Differences 
Historically, all enrollees have been given equal weight in rating per- 
formance, whatever the measure used. The findings in this report and 
similar ones from other studies, however, suggest that consideration be 
given to the development of performance formulae that allow outcome stand- 
ards to vary by local econanic conditions, registrant characteristics, and 
even by service components. Regression adjustment to control for these 
variations, as is done in the JTPA formulae, has the advantage of 
permitting more t*-Xible performance standards for programs serving groups 
with a lew likelihood of finding employment readily or high service costs, 
or programs operating in labor markets where it is hard to find jobs. 

But regression formulae also have disadvantages. The formulae can be 

complex, making them unsuitable for communication of program objectives to 

« 

local staff or for setting performance criteria for service subcontractors. 
Moreover, the regression procedure must be properly executed: a formula 
must include the most important determinants of outcomes and must be 
estimated from an appropriate sample if its message is to be correct. 

This study presents some simpler formulae, which weight individual 
performance indicators by prior employment and prior welfare receipt (on 
the basis of the subgroup impact analysis), the characteristics used to 
def.lne the major subgroups. These measures were chosen because they are 
both important predictors of future behavior for adult WIN-mandatory AFDC 
wcraen, as this report will show. This choice does not imply that other 
approaches have been rejected. The study recognizes that no single 
approach should necessarily be applied to all programs and labor markets. 
It recognizes that the goals of sane programs may not be easily translated 
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into any single formula, whether simple or complex. Moreover, different 
goals may require different formulae. 

II. The Demonstration of State Work/Welfare Initiatives 

MDRC's Demonstration of State Work/Welfare Initiative.^ was launched in 
1982 to test the effectiveness of state employment programs for people 
applying for or receiving AFDC. For the most part, states were using their 
new authority to experiment with WIN program variations authorized by the 
Omnibus Budget Reconciliation Act (OBRA) of 1981. The MDRC study includes 
programs in 11 states, eight of which used random assignment to form 
experimental and control groups fc full-scale impact and benefit-cost 
studies. Most programs set the goals of increasing employment and reducing 
the dependency of the welfare population by preparing recipients for work. 
They required most able-bodied recipients to participate in job search 
and/or unpaid work experience or other activities as a condition of welfare 
receipt. 

The research was designed to assess three aroas: the feasibility of 
implementing a mandatory participation and/or work requirement; the 
programs' impacts on employment, earnings and welfare receipt; and the 
cost-effectiveness of the different approaches. Findings from this MDRr 
demonstration are being released as the results for each state 1 s program 
become available. The five initiatives included in this subgroup study are 
described in Chapter 2. 

Among the five program evaluations on which this report is based, four 
found positive employment and earnings impacts. Four of the five also 
obtained short-run welfare reductions. And four of the five indicated that 
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the initial investment of program funds would result in net government 
budget savings in five years or less. 

The individual evaluations included sane research on subgroup impacts. 
Those findings suggest that programs may indeed have greater impacts for 
some groups within the diverse welfare population than for others. For 
example, employment increases have often been larger for clients without a 
recent work history than for those who have worked during the year prior to 
program enrollment. These findings are buttressed by KDRC research on 
prior WIN programs and the results from the National Supported Work 
Demonstration. 11 The study reported on here is able to examine a wider 
variety of subgroups than were analyzed in the individual evaluations. It 
also uses longer-term data and a methodology more suited to che issue of 
performance measures. 
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CHAPTER 2 



THE PROGRAMS AND PARTICIPANTS INCLUDED IN THE STUDY, AND 
THE NORMAL DYNAMICS OF EARNINGS AND WELFARE RECEIPT 

To provide some context, for the discussion of program impacts in 
Chapter 4, this chapter describes the programs included in the evaluation, 
the characteristics of the registrants who were offered \he opportunity to 
participate, and the welfare and employment experiences they .fould have had 
in the absence of program services. The first section discusses key 
similarities and differences smong the programs included in the subgroup 
study: San Diego, Baltimore, Virginia, Arkansas, and Cook County, Illinois 
(which includes the City of Chicago). The second section describes the 
characteristics of sample members. The third section looks at the earnings 
and welfare receipt patterns for different subgroups of controls. . The 
fourth section discusses hew the major subgroups are defined for the 
analysis. The final section illustrates the differences in program costs 
for different subgroups by comparing costs of the San Diego and Baltimore 
programs. 

I . The Program Mode ls 

No single program model was tested in MDRC's Work/Welfare study. 
Rather, the participating states implemented their own initiatives, using 
different strategies. Characteristics of the local WIN-mandatory popula- 
tions olten differed as well. 

The evaluations, on the other hand, are similar in methodology: Each 
study used an experimental design whereby program eligibles were randomly 
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assigned to one or more experimental groups or to a control group. Members 
of the experimental groups were required to take part in the program 
services being evaluated, whereas the control groups were barred from the 
special program services, although in seme areas they could receive the 
minimal services offered under the regular WIN program. Data were collect- 
ed on participation measures, employment and welfare outcomes, and direct 
program operating costs. To estimate program impacts, the employment and 
welfare behavior of the experimental and control groups were compared over 
several quarters of follow-up. Because random assignment produced experi- 
mental and control groups with similar demoyraphic characteristics and 
backgrounds in prior employment and welfare dependency, any statistically 
significant differences in behavior during the follow-up period can be 
confidently attributed to the program. 

The term applicant identifies a person applying for AFDC at the time 
of entry into the research sample, whether or not. that person's welfare 
grant was subsequently approved. That label remains, even for those 
applicants who never get approved, and even for those applicants who become 
recipients. The terat r ecipient refers to a sample member who was already 
receiving welfare at the date- of sample entry. These two subgroups are 
important and are analyzed separately throughout th± study. Other 
subgroup divisions are based on prior demographic and background 
characteristics. 

Table 2.1 shows the key characteristics of the programs included in 
this analysis. Table 2.2 shows rates of participation for experimentals in 
the various components. Length of follow-up is important for interpreting 
participation rates. In these samples the follcw-up for participation is 
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different from the follow-up for the impact data and is shown for each 
program in Me bottom rcw of the table. The published state reports 
contain more detail about both the programs and the evaluation results. 1 ' 2 

Briefly, job search and work experience were the major program 
services; but states differed in the mix and intensity of these services, 
their sequencing, and the populations that received them. Some parti- 
cipation in „ation and traini'.g was recorded for Baltimore, Virginia, 
and Cook County. But in the latter two this participation largely 
reflected referrals to outside providers or self-initiated activity and was 
little or no higher than the background levels observed for controls. The 
proportion participating in any program activity varied from 38 percent in 
Arkansas to 58 percent in Virginia. The programs were all mandatory, but 
differed in the extent to which participation was enforced and the degree 
to which monetary sanctions were used as a tool of enforcement. The 
proportion sanctioned varied from practically zero in Baltimore to 12 
percent in Cook County. 

San Diego worked with all WIN-raandatory welfare applicants but did not 

enroll recipients. Expeiimentals went through a two-stage fixed sequence 

of group job search, followed by a 13-week work obligation if they had not 

3 

found unsubsidized jobs in the first phase. San Diego's decision to focus 
entirely on applicants represents one targeting option available to program 
operators. 

Baltimore enrolled both WIN-mandatory applicants and recipients, but 
only recipients who had just become mandatory, usually because their 
youngest child had turned six years of age. The program provided a mix of 
components (including job search, unpaid work experience, education and 
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TABLE 2.1 



KEY CHARACTERISTICS OF STATE WORK/WELFARE INITIATIVES 



Chorocterlstic 


T 

Son Diego, Col Ifornlo 


1 r 

Boitlmore, Morylond 


E! Iglble Group 
Appl iconts 

Newly Mondotory Recipients 
Currently Mondotory 
Recipients 


Yes 
No 

No 


Yes 
yes 

No 


Enrol Iment Limit 


None 


1000/yeor 


Progrom Model 


Job Seorch WOrkShnn fnl Invert 

by 13 weeks of CWEP In publ Ic 
ond prlvote nonprofit 
ogenc les . 


mui t i -component , incl ud Ing 
1 ob seorch, educotlon, troln- 
lng, on-the-job trolnlng ond 
10 weeKs ot work experience. 


Sequence 


Fixed: job seorch then 
work experience 


Discretlono ry 


Client Choice of Components 


No 


Yes 


Components 
Job Seorch 

independent 
Group 


Mondotory 

No 
Yes 


Mondotory when Judged appro- 
priate 

yes 
Yes 


Work Experience 


Mondotory If no |ob found 
through job seorch 


Mondotory when judged appro- 
priate 


EducotMn nnd Trolnlng 


Not offered 


In-house ond by referral 


Study Areo C 


County-wide 


10 out of the 18 Income 
Molntenonce Offices 


Control Services 


WIN services 


WIN services 


Somple Enrollment Period 


October 1982 - August 1983 


November 1982 - December 1983 
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TABLE 2.1 (continued) 



. . 0 

Virginlo 


F b 
Arkonsos 


Cook Cr ty, Illinois 0 


Yes 
Yes 


Yes 
Yes 


Yes d 
Yes 


Yes 


Yes 6 


Yes 6 


None 


None 


None 


Job seorch foi lowed by |3 
weeks of CWEP, educollon or 
trolnlng. 


Job seorch workshop followed 
by Indlvlduol fob seorch ond 
12 weeks of work experience 
In public ond prlvote non- 
profit ogencles. 


Indlvlduol fob seorch followed 
by 13 weeks of CWEP In privote 
non-profit ogencles. 


job seorch first 


Job seorch first 


Job seorch first 


Yes, ofter |ob seorch 


No 


NO 


Mondotory os first component 


Mondotory 


Mondotory 


Yes 
Yes 


Yes 
Yes 


Yes 
No 


Mondotory when |udged oppro- 
prlote 


Mondotory, but used 
Infrequently 


Mondotory If no |ob found 
through |ob seorch 


By referrol 


Not offered 


Not offered 


11 of 124 ogencles (4 urbon, 
7 rurol) 


PuloskI South ond Jefferson 
Counties 


16 out of the 22 Income 
Molntenonce Offices 


No speclol services 


No speclol services 


Attonrt the orlentotlon session 


August 1983 - September 1984 


June 1983 - Morch 1984 


Februory 1985 - November 1985 



NOTES: °ln Son Diego, VI rg Into, ond Cook County, there ore two different experlmentol 
treotments. In Virginlo, the two experlmentol groups were merged for the onolysls. 

b |n Morylond ond Arkonsos, o full evoluotlon wos conducted In the Indicated 
counties ond o process study covered other oreos os well. 

c |n addition to the study oreos, Virginlo ond Illinois Implemented their programs 
stotewlde ond Arkonsos ond Morylond In selected oreos. 



d Unl Ike other stotes, oppllconts In Cook County were oil opproved for AFDC before 



enrollment. 

e. 



Although 'currently mondotory recipients* were eligible for the progrom, this 
group wos not Included in the research sample. 
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TABLE 2.2 



DEMONSTRATION OF STATE WORK/WELFARE INITIATIVES: 
PARTICIPATION AMONG AFDC EXPERIMENT ALS 



Progrom Activity Meosure 


Son Dlegc 


Baltimore 


Vlrglnlo 0 


Arkonsas 


Cook County 


Portlc Ipotion Rote 












Any Activity (X) 


44.6 


45.0 


58.3 


38.0 


47.2 


Job Seorch 












lndlvlduol (X) 


— 


n 


40.4 


23.3 


36.1 






J24.7 








Group (X) 


42.3 




14.7 


27.3 


— 


Work Experience (X) 


11.8 


17.5 


9.5 


2.9 


7.3 


Faction ^nd Trolnlng C (X) 


4.1 


17.3 


11.6 


2.4 


16.9 


Dereglstered " ' 


52.1 


37.6 


42.3 


57.5 


56.9 


Due to Request for 












Sanctioning (X) 


6.6 


rore 


3.8 


4.3 


12.4 


Sample Size 


1540 


1362 


2138 


245 


4050 


Follow-Up Period In Months 


Six 


Twelve 


Nine 


Nine 


Nine 



SOURCE: MDRC Work/Welfore pemonstrotlon Reports. 

NOTES: °For Virginia, activity meosures ore bosed on both experimental groups, 
which differed In Intenoed occfcss to worh experience ond education ond training 
octlvltles. 

b For Son Diego end Cook County, activity meosures refer *o post-registration 
or post-orlentotlon octlvltles. 

c 1 ncl udes services other thon education ond training In San Diego ond 

Arkonsos. 
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training), and staff made service assignments taking into account 
enrollees ' needs and preferences, depending on their assessments and the 
availability of open slots. In order to ensure adequate funding on an 
individual basis for this somewhat broader array of services than offered 
in the other programs, the Baltimore p* >grara restricted active enrollment 
to only 1,000 registrants per year during the period studied. 

Virginia enrolled its entire WIN-mandatory caseload. The state 
stipulated that counties require job search of all enrollees but author- 
ized, as a county option, short-term work experience, education and train- 
ing as follow-up activities. Education and training were not provided by 
the program; rather, participants were referred to JTPA and community 
schools with independent funding, open to all who qualified. As it turned 
out, enrollees participated in education and training at the same rate as 
controls. 

Arkansas enrolled its entire WIN-mandatory caseload during the program 
start-up phase, but only applicants and recipients who became mandatory 
after the research began were included in the impact Scjnple. The program 
consisted primarily of independent and group job search and, less 
frequently, a work experience component. ThicC features distinguish the 
Arkansas sample. First, the state has relatively low AFDC grant levels. 
Individuals are therefore likely to apply for AFDC if they have very little 
opportunity for income through work or through other f an, -ly members. As a 
consequence, employment rates — bo'.h before and after enrollment — were 
lower for this sample than for the others. Second, Arkansas had the 
largest share of applicants whoso grants were not approved. For this 
reason alone welfare receipt will be lower for this sample than for other 



samples included in the study . 

Third, Arkansas, under federal waiver provisions, filed for and 
received permission to classify as WIN-mandatory AFDC case heads whose 
youngest child was aged three to five years. As stated in Chapter 1, much 
of the descriptive analysis of potential target groups has focused on women 
with children younger than school age. The Arkansas sample affords an 
opportunity to examine actual program effectiveness on an important part of 
this subgroup, albeit with a small sample and at grant levels below those 
of most states. 

The Cook County program only worked with recipients and a subset of 
applicants, those whose AFDC grants had already been approved. As with 
Arkansas, the Cook County research sample was restricted to those who 
became WIN-mandatory after the research began. Sample members were expect- 
ed to participate in independent job search for two months. They were 
required to make 40 employer contacts per month and to report on these at 
biweekly group sessions. A stock of short-term unpaid work experience 
positions was also maintained, and individuals who completed job search 
without a job could be assigned to a worksite . when one opened. 4 In 
addition to these program activities, experimentals were allowed to 
participate in education and training activities they might find on their 
own. Finally, compared to other programs, staff were more o_ lented towards 
obtaining welfare reductions from reported client employment and sanction- 
ing e ollees who failed to carry through on their assignments than toward 
service provision. 

These five programs were all relatively inexpensive, although they 
varied somewhat in average cost. For example, the net cost of the San 
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Diego program (in 1987 dollars) was about two-thirds that of the Baltimore 
program — which spent, on average, $1,050 per experimental. Costs for 
Virginia (around $450 per experimental) were in the middle of the range for 
the five programs. Arkansas and Cook County cost the least, at $170 and 
$150 per experimental, respectively. Programs also differed in vhere they 
chose to concentrate resources. San Diego spent more on ensuring 
compliance with its participation requirement (which entailed monitoring, 
registrant follow-up and limited sanctioning), for example, whereas 
Baltimore offered more expensive services, such as education and training, 
and provided client stipends. 

The programs also differed in the proportion of eligibles covered. 
This is different from the proportion that participated because it also 
includes those who became employed, lef : welfare for other reasons, or were 
sanctioned. People were defined as not covered by the program if they were 
still on welfare, not employed, and had not participated in a program 
activity nine months after program entry. In San Diego, for example, the 
participation rate was 45 percent, uut only 10 percent were not covered by 
the program within nine months of enrollment. This high San Diego coverage 
indicates that a short-term participation requirement was, in fact, 
realized by that program. In contrast, a larger proportion of continuing 
registrants — almost one quarter — were not covered in the Baltimore 
program. This may be due partly to inclusion of new WIN mandatory 
recipients in Baltimore and the staff's greater flexibility in deferring 
registrants from activities. In Virginia, most experimentals were covered 
by the program (nearly 90 percent), but the minimum requirement — a 
loosely structured form of independent Job search — was relatively easy 
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for both the program and the clients to fulfill. Arkansas and Cook County, 
the two programs with lowest costs per enrollee, had different coverage 
rates. Participation rates were somewhat lower in Arkansas than else- 
where, but approval rates for applicants were also lower, yielding an 
overall coverage rate that was similar to Baltimore's. Cook County, 
despite its focus on recipients and despite its large caseload, reached 94 
percent coverage by using independent job search as the main component, 
backed by routine sanctioning for noncompliance. 

Statutory grant maximums, based on state standards of need, varied 
widely, requiring extra care in making comparisons among programs. Low 
benefit levels increased the attractiveness of low-wage jobs in sane areas, 
and also increased the likelihood of a case closure when employment was 
obtained. Local economic conditions, staff experience and attitudes also 
differed. In San Diego, welfare recipients had a relatively good market in 
which to look for jobs, but in rural areas of Virginia, the prospects for 
employment were more limited. And, in the administrative reorganization 
permitted by OBRA, social service staffs in some states ~ those who had 
not had responsibility for employment functions under the previous system 
— had to go through a learning process. However, staffs in San Diego and 
Baltimore had substantial prior experience in operating such programs, 
which contributed to their programs' smooth administration. 

As shewn in Table 2.1, services available to controls varied across 
experiments. In Virginia and Arkansas, no special services were provided. 
In Cook County, controls were required to attend only the initial orienta- 
tion meeting, but were not assigned to job search or work experience. In 
San Diego and Baltimore, controls were assigned to the existing WIN 
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programs, although services there were low* For example, participation 
rates among Baltimore controls were under 5 percent* 



II* Sample Characteristics 

The sizes and demographic composition of the research samples for the 
five programs analyzed in this report are shown in Table 2*3. 5 The evident 
variation is due to differences in the program models, targeting 
philosophies, and environments in which the programs operated* As noted 
earlier, each program served the WIN-mandatory caseload or portions of that 
caseload* In four of the five programs, this meant excluding most women 
with children less than si . years old* Arkansas extended WIN-mandatory 
status to AFDC women whose youngest child was aged three to five* The San 
Diego program served only applicants, while the Baltimore, Virginia, and 
Arkansas samples had a fairly even mix of applicants and recipients, 
although the type of recipient differed* Cook County worked with mandatory 
applicants and recipients, but only applicants whose AFDC grant had been 
approved* 

There were other differences not shown in Table 2*3* AFDC approval 
rates for Arkansas applicants were lower than elsewhere, for example, 
partly because program enrollment occurred at the time of the welfare 
application rather than later* Also, as indicated earlier, employment 
rates in Arkansas were lew compared to the other states, probably 
reflecting the lew grant levels there* 

Analysis of the Cook County data proved to be the most difficult to 
integrate with the oth" r programs* Applicant status was not available in 
the data set as it was in other states and had to be inferred from the 
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TABLE 2.3 



SELECTED CHARACTERISTICS OF AFDC 
APPLICANTS AND RECIPIENTS AT TIME OF RANDOM ASSIGNMENT, 
BY PROGRAM AND WELFARE STATUS 





Son Diego 


Bol t imore 


Virginio 


Subgroup 


Appl iconts 


Appl iconts 


Recipients 


TotOl 


Appl iconts 


Recipients 


Totol 


Reseorch Group [%) 
Experimentol 
Cont rol 
Other 


46.5 
27.0 
26.5 


48.6 
51.4 


50.2 
49.8 


49.4 
50.6 


66.7 
33.3 


67.7 
32.3 


67.3 
32.7 


Prior Eornings [%) 
$3000 or More 
5l-2yyy 
None 


28.8 
22.9 
48.4 


31.9 
29.3 

JO • 0 


6.7 
20.6 
72 . o 


19.3 
24.9 
55.8 


29.2 
28.4 
42.3 


3.3 
19.7 
77.0 


13.7 
23.2 
63.0 


Hod own AFDC Cose (X) 
Never 

Two Yeors or Less 
More Thon Two Yeors 


33.4 
38./ 
27.9 


22.7 

4 1 R 
t l . 0 

35.5 


5.2 

L 1 • 1 

73.8 


14.0 
31.4 
54.6 


26.2 
31 .7 
42.2 


2.5 
25.7 
71.8 


12.0 
28.1 
59.8 


High School D i p I onio (%) 
Yes 
No 


61.5 
38.5 


44.9 


42.1 

C7 0 

J/ . 7 


43.5 

C L C 
36. 3 


50.8 
49.2 


38. 8 
61.2 


43.6 
56.4 


Child 12 or Under {%) 
No 
Yes 


22.6 
77.4 


27.5 


13.4 
86 • 6 


20.5 

/y. 5 


22.9 
77.1 


23.7 
76.3 


23.4 
76.6 


Number of Own 
Children [%) 
One 

More Thon One 


49.7 
50.3 


49.6 


A7 1 

56.9 


46.0 

53.2 


4y.6 
50.4 


42.0 
58.0 


45.0 
55.0 


Currently Morried [%) 
Yes 
No 


46.6 
53.4 


49.6 


Js . J 

65.7 


4 Z.J 

57.7 


49.3 
50.7 


38.3 
61.7 


42.8 
57.2 


Ever Morried (%) 
Yes 
No 


84.1 
15.9 


69.9 
^n i 


49.1 
en o 

DU. 7 


59.5 
4U. 3 


74.2 
25.8 


65.3 
34.7 


68.9 
31.1 


Age (%) 
30 or Over 
Under 30 


65.6 
34.4 


65.4 
34 .6 


42.7 
57 1 


54.0 
** o • u 


64.0 

Jo • U 


65.9 
34 • 1 


65.1 
34 .9 


Ethnicity [%) c 
white 
Block 
Hisponic 


61.5 
20.7 
17.8 


33.8 
66. 2 


25.1 

74 9 

/ *i • 7 


29.5 
7n k 

/ U . 3 


41.8 

CO o 
30.2 


26.8 
73. / 


32.8 
67.2 


Recent Ul Benefits [%) 
Some 
None 


14.1 

QC 0 
CO. 7 








3.9 
96.1 


0.4 
99.6 


1.0 
98.2 


Lobor Morket {%) 
Urbon 
Rurol 










78.7 
21.3 


78.8 
21.2 


78.8 
21.2 


Somple Size** 


3238 


1380 


1377 


2757 


1269 


1881 


3150 



(continued) 
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TABLE 2.3 (continued) 





Arkonsos 


Cook County 0 ' b 


Subgroup 


Appl iconts 


Recipients 


TotOl 


Appl iconts 


Rec ipients 


TotOl 


Reseorch Group [%) 
Experimentol 
Control 
Other 


50.0 
50.0 


49.2 
50.8 


49.7 
50.3 


34.5 
32.0 
33.5 


33.8 
31.9 
34.3 


34.0 
31.9 
34.1 


Prior Eornings [%) 
$3000 or More 

WWW wl fflw 1 V 

$1-2999 
None 


12.5 
22.5 
64.9 


1.3 
7.2 
91.5 


8.0 
16.3 
75.7 


33.0 
20.4 
46.5 


4.9 
14.9 
80.2 


14.4 
16.8 
68.8 


Hod Own AFDC Cose (%) 
Never 

Two Yeors or Less 
More Thon Two Yeors 


56.2 
36.7 
7.0 


8.1 
26.9 
65.0 


36.7 
32.7 
30.5 








High School Dipiomo [%) 
Yes 
No 


55.1 
44.9 


42.0 
58.0 


49.8 
50.2 


46.8 
53.2 


30.9 
69.1 


36.2 
63.8 


Chi Id 12 or Under 1%) 
NO 
Yes 


9.6 
90.4 


11.8 
88.2 


10.5 
89.5 








Number of Own 
Children {%) 
One 

More Thon One 


42.5 
57.5 


34.8 
65.2 


39.4 
60.6 


— 


— 


— 


Currently Morried [%) 
Yes 
No 


31 .V 
66.1 


21.2 
78.8 


27.6 
72.4 


— 


— 


— 


Ever Morried (%) 
Yes 
No 


56.9 
43.1 


41.8 
58.2 


50.8 
49.2 








Age (%) 
30 or Over 
Under 30 


38.5 
61.5 


38.1 
61.9 


38.3 
61.7 


62.1 
37.9 


43.1 
56.9 


49.5 
50.5 


^hnicitv fX) c 

bl Mil 1 V> 1 1 y \™ i 

White 
Block 
Hisponic 


16.7 
83.3 


8.5 
91.5 


13.4 
86.6 


21.8 
65.5 
12.6 


14.3 
75.3 
10.4 


16.8 
72.0 
11.1 


Recent Ul Penef its [%) 
Some 
None 














Lobor Morket [%) 
Urbon 
Rurol 


64.8 
35.2 


56.5 
43.5 


61.4 
38.6 








Somple Size** 


670 


457 


1127 


4014 


7898 


11912 



SOURCE: Demogrophic information is from MDRC Client Information Sheets. Prior eornings were 
coiculoted from the County of Son Diego Unemployment Insuronce records from the EPP Information System; 
from the Commonweal th of Virginio Unemployment Insuronce eornings records; from State of Arkonsos 
Unemployment Insuronce records; ond from the Illinois Unemployment system eornings records. 
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Toble 2.3 (conlinued) 



nnnl ."°l S: . Distributions may not odd to exoctly 100.0 percent due to rounding. Coteqories not 
!efe "o! ?o!culS?e r d! C pr ° 9r ° m S ° mp ' eS ° re indi " t6d Wi ' h 0 d0Sh ' Tests of stotiSTlc^ signKconce 

MID system record s^ 0 ""^ dem ° 9r0 P hic infor ™tion is from the Stote of Illinois Deportment of Public Aid 

progroms. See^^^or^^uss^on; 0 "' iC ° nr ' " - * reci P ienr ' f3r c °<* ^ from the other 

irsM- ia . t CFor Bo,rimore ' Virginia, and Arkonsos the category 'block" includes o smoll number of 

is2: w b !S^: s „^!rp;n?rro^ f .?f e oop P -=o.^. son 0,630 °°° t * ^ inc,wes ° « 

^ r\ 21 I 0 0 to™, 0 * 3238 t for Son Diego ond 11912 for Cook County. Impoct calculations do not 
include Job Seorch Only Experimental s; the sompie sizes for Son Diego and Cook County in the impoct 
onolysis ore, therefore, 2381 ond 7855, respectively. Y W 
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absence of AFDC payments prior to random assignment* in addition, the fact 
that only individuals whose AFDC applications had already been approved 
were enrolled in the program meant that ■applicants" were more likely to be 
on welfare during follow-up than applicants in the other samples. More 
sorious, information about the length of prior welfare receipt and other 
characteristics for Cook County is lacking because th'a information sheets 
used to obtain subgroup characteristics did not cover the full impact 
sample in Cook County. 

The Baltimore and Virginia samples were similar in many respects: over 
half had neither a high school diploma nor a GED; more than half had been 
receiving AFDC for more than two years; and, on average, only about 40 
percent had held a job in the year prior to random assignment. The San 
Diego sample was less disadvantaged: more than half were high school 
graduates; less than 30 percent had been on welfare for more than two 
years; and one-half had held a job in the year before this welfare 
application. 

Ethnic composition also differed. In Baltimore and Virginia, between 
60 and 70 percent of the samples were black; in Arkansas, over 85 percent 
were b-ack; in San Diego only 20 percent of sample members were black, with 
Hispanics making up 18 percent of the sample. in Cook County, more than 
two-thirds of the sample members were black, about 11 percent were 
Hispanic. Data on educational attainment in Cook County, not strictly 
comparable to Uie data for the other programs because they came from case 
records rather than direct interview, indicate that nearly two-thirds of 
the sample did not hava a high school diploma. Arkansas was characterized 
by particularly lew rates of prior-year employment, under 25 percent. In 
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addition, prior welfare histories among Arkansas sample members varied 
widely, with about a third of the sample never having had an AFDC case in 
the past and a third having had a case for more than two years. 

Comparisons of applicants and recipients reveal large differences in 
prior earnings and prior welfare receipt in all programs. Applicants not 
only had shorter welfare histories, but also had more recent earnings and, 
except for Baltimore, more education. Recipients were more likely to be 
unmarried at enrollment and also more likely never to have been married. 
At least two-thirds of recipients in the three samples with data had a 
welfare history of more than two years. 

III. Earnings and Welfare Receipt: The Normal Dynamics 

A wide range of earnings and welfare information on WIN-mandatory 
clients in the absence of special jogram intervention can be captured by 
simple objective measures, collected as part of the program enrollment 
process and readily verifiable. As an illustration, Figures 2.1 and 2.2 
plot the earnings and welfare receipt of the early Baltimore control sample 
over the three- year period after random assignment, for selected subgroups 
defined by applicant/recipient status, length of prior welfare receipt, and 
prior earnings. The Baltimore sample was selected because it has the full 
spectrum of applicant and recipient subgroups and a long enough follow-up 
period to show the importance of changes in status over time. 

The subgroup differences are typically large. Quarterly average 
earnings for control group applicants without a prior welfare history and 
with $3,000 or more in earnings in the year prior to AFDC application 
(i.e., the point of random assignment), for example, fall into the $1,200 



-29- 

\ * * 



FIGURE 2. 1 
BALTIMORE CONTROLS. EARLY COHORT: 
QUARTERLY AVERAGE EARNINGS, BY SUBGROUP 
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Applicants, Two Years or More 
Prior AFOC, No Prior Earnings 
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FIGURE 2.2 
BALTIMORE CONTROLS, EARLY COHORT: 
QUARTERLY AVERAGE AFDC PAYMENTS. BY SUBGROUP 

AvGrage AFDC Payments ($) 
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to $1,800 per quarter range over the follow-up period (these estimates 
count zero earnings for persons not employed). During the same period, 
subgroups with no recent employment history and a pattern of AFDC receipt 
for more than two years never reached average earnings over $400 per 
quarter. 

The pattern of differences can change over time. In the case shown, 
subgroups at the top and the two subgroups at the bottom remained 
relatively stable, but the groups in between showed upward trends. 

Differences in welfare payments across subgroups are also large. All 
subgroups show welfare receipt declining over time. But at the end of the 
12 follcw-up quarters, those recipients with two cr more years of welfare 
receipt and no pre-program earnings were receiving three to four times the 
quarterly benefit payments of first-time applicants. 

As can be seen, the majority of welfare receipt is accounted for by 
a minority of AFDC women. For example, in the Baltimore control sample, 
applicants constituted about one-half the sample but were consuming less 
than one-third of the AFDC expenditures on the control group at the 
three-year mark. At the other end of the spectrum, recipients of more than 
two years standing who had no earnings in the pre-program year were only 
one-third of the sample, but were receiving nearly half of the welfare 
payments to the control sample at the end of three years. 6 A further 
breakdown of recipients by whether or not they had high school diplomas 
revealed that dropouts who had not obtained even a GED comprised 18 percent 
of the cample but received 28 percent of the AFDC dojlars. 
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IV. Subgroup Differences without Intervention 

There are two complementary approaches to defining and studying 
subgroups. The first is to disaggregate the sample by specific individual 
characteristics, as is done in Table 2.2, and then define persons with 
certain combinations of characteristics as constituting a subgroup (such as 
recipients with no pre-program earnings and no high school diploma). 
Analysis of subgroups defined in this manner may provide considerable 
irformation about the potential results of highly specific targeting 
possibilities. These can be loosely linked to more general concepts of job 
readiness or welfare dependency, ^his study defines job ready as likely to 
become employed during the follow-up period. It defines dependent as 
likely to remain on AFDC during follow-up. The alternative approach uses 
all the individual characteristics in a multiple regression framework to 
predict job readiness and dependency. Subgroups are then categorized on 
the basis of similar predicted job readiness or dependency. 

This report emphasizes the first approach. Table 2.4 shows differ- 
ences in employment and welfare receipt rates during follow-up for several 
important subgroups of this study, pooling the data for controls in San 
Diego, Baltimore, Virginia, and Arkansas. These rates for controls again 
depict normal behavior (i.e., in the absence of the special program being 
evaluated). Quarter 6 was chosen as the end point because it is the last 
quarter for which all sample members had data. Data for Cook County 
controls were not included, because subgroup definitions for the impact 
sample for that program are not precisely comparable to the v *f initions for 
the other programs. The table shows the average levels of employment and 
welfare recej.pt of persons with a particular characteristic. 
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TABLE 2*4 



AFDC CONTROLS : EMPLOYMENT AND WELFARE 
RECEIPT BY MAJOR SUBGROUP 



Subgroup Cho roc te r i s t i c 


Averoge Quorterly 
Empl oyment Rote, 
Ouorters 4-6 (%) 


Percent Receiving 
Any AFDC Poyrnent, 
Quorter 6 [%) 


Welfore Stotus 
Appl icont 
Rec ip i en t 


3 7*0 

9 9 7 




41.6 
72.4 


Prior Yeor Eornings 
13000 or More 
$1-2999 
None 


60*0 
4 3*0 
17*2 




38*5 
4 9*3 
60*4 


Hod Own AFDC Cose 
Neve r 

Two Yeors or Less 
More Thon Two Yeors 


J 6 * J 

36*5 
25*7 




3 4*8 
46*4 
67*1 


Hiyh School Diplomo 
Yes 
No 


38*0 
24 *9 




47*1 
60*2 


Child 12 or Under 
No 
Yes 


31*3 
31 .4 




43*3 
56.4 


Number of Own Children 
One 

More Thon One 


32.7 
30*2 




51.7 
55.5 


Currentl y Morr ied 
Yes 
No 


32*3 
30*7 




4 7.4 
58. C 


Ever Married 
Yes 
No 


32*6 
29*1 




47.0 
66.3 


Age 

30 or Over 
Under 30 


32.2 
30*2 




49.6 
59.4 


EthnL j° 
Wh i 

Bl OCrv 

Hisponi c 


33*6 
29*4 
43.9 




39.2 
62.4 
42.1 



SOURCE: MDRC colculotions from MDRC Client Information Sheets; from the 
County of Son Diego welfore records ond Unemployment Insuronce records from the 
EPP Informotion System; from the State of Maryland welfore ond Unemployment 
Insuronce records; from the Commonwealth of Virginia Unemployment Ins-^once 
eornings records, welfore records from the Virginia Automated Client .formotion 
System, ond Foirfox County AFDC cose files; ond from Stote of Arkonsu welfore 
ond Unemp I oyment fosuronce records. 

NOTES: Toble entries were estimated from control samples pooled ocross 
Son Diego, Boltimore, Virginia, ond Arkonsos* Estimates ore not regression 
od j us ted * 

°For Boltimore, Virginia, ond Arkonsos the category 'block' 
includes o smoll number of individuals in other non-white groups. In Son Diego 
ond Cook County, 'white' includes o small number of tion-block, non-h i s pon i c , 
non-wh ite pe rsons. 
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It is important tv note that these comparisons do not account for 
differences among individuals in any other characteristic. Regression 
analysis can take these differences into account. Table 2.5 shows the 
effects on employment and welfare receipt that are properly attributable to 
particular characteristics when differences attributable to the other 
characteristics have been accounted for. 

It is useful to explain briefly how to interpret Table 2.5. For each 
characteristic or range of character istics, the number she a in the table 
indicates the additional effect of the characteristic relative to some 
benchmark. Take the effect of being an AFDC recipient, for example. The 
fifth row .'.own in the right-hand column indicates that being an AFDC 
applicant reduces by 16 percentage points the likelihood of receivin v an 
AFDC payment in quarter 6, relative to the likelihood of a recipient 
receiving AFDC in quarter 6. The effect of prior earnings experience is 
interpret able as follows (see the seventh, eighth, and ninth entries in the 
left-hand column): Earning $3,000 cr more ir. the year prior to random 
assignment adds 38.8 percentage points to the likelihood of being e^nloyed 
in quarter 6, relative to the likelihood of being employed in quarter 3 for 
someone with no earnings in the year prior to randan assignment. Earning 
between $1 and $2,999 adds 23*8 percentage points to the likelihood of 
being employed in quarter 6, also relative to the likelihood for someone 
having no earnings in the year prior to randan assignment. The increased 
likelihood of being employed in quarter 6 for those with higher earnings 
versus those with lower but still positive earnings in the prior year is 15 
percentage points (38.8 minus 23.8). 

As is clear from the tables, in these samples of adult enrollees, 
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TABLE 2.S 



AFDC CONTROLS: EFFECTS OF SUBGROUP CHARACTERISTICS ON 
EMPLOYMENT AND WELFARE RECEIPT 



Subgroup Cho r oc t er i s t i c 


Averoge Quorterly 
Empl oyment Rote, 
Quorters 4-6 [%) 


Percent Receiving 
Any AFDC Poyment, 
porter 6 [%) 


Progrom Loco 1 i on 
Sor Diego 
Bo 1 t i more 
Virginia 


- 3.4* 
+ 3.1** 

- 1 Z ♦ Z * * * 


- 8.0*** 
-15.6*** 


AFDC Status 
App! icont 
Rec ipient 


+ 2.0 


-16.0*** 


Prior Yeor Earnings 
$3000 or More 
$1-2999 
None 


+38.8*** 
+23.8*** 


-11.1*** 
- 5.8*** 


Hod Own AFDC Cose 
Never . 

Two Yeors or Less 
More Thon Two Yeors 


- 3.1* 

- 6.4*** 


+ 7.0*** 
+16.2*** 


High School D i p 1 omc 
or Equivolent 

Yes 

No 


+ 8.1*** 


- 8.8*** 


Youngest Chi Id Age 
Over 12 Yeors 
6-12 

Less Thon 6° 


+ 0.4 
+ 3.9* 


+ 9.6*** 
+ 5.3** 


Number of Own 
Chi 1 oren 
One 

U A r D Thnn fin o 
ffiui t i nun Ullc 


T U ♦ J 


+ 9 1 


Age 

30 or Over 
25-30 

lace Thnn 9 £ 

L«) ) i nun lj 


+ 0.1 

- t R 


+ 0.7 


t ve r mO r i I cO 
Yes 
No 


4. 0.5 


+ 7 .2*** 


Ethnicity b 
White 
Bl ock 
Hi spon ic 


- 0.5 
+ 9.0*** 


+13.5*** 
+ 8.3** 


Cons tan t 


+17.8*** 


+48.9*** 


Unodjusted R 2 


.2060 


.1774 



SOURCE: Sv;C Toble 2.4. 

NOTES: Toble entries ore coefficients from o regression run on control 
sompies pooled across Son Diego, Baltimore, Virginia, and Arkansas. Somple size 
"s 3869. Regression coefficients ore olwoys estimated os differences relotive 
\o o reference cotegory. The reference cgtegor es ore indicated w>th doshes. 
For exomple, being on AFDC opplicont leods to o quorterly overage employment 
rote thot is 2 percentage points higher thon the rote for AFDC recipients. 
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TABLE 2.5 (continued) 



hnn rK-.,<m* , A* P[°9 ro *' s except Arkonsos obout 10 percent of the sompies 
hod children less thon 6 yeors old. In Arkonsos 50 percent hod children less 
iinpocts ° n Arkonsos subgroup is broken out for seporote onoiysis of 

■ . . ^For B °l f ™re, Virginio, ond Arkonsos the cotegory 'block* 

mciuoes o smoll numbei jf indivi uols in other non-white groups. in Son Diego 
ond Cook County, 'white" includes o smoll number of non-block, non-hisponic , 
non-wh i te pe rsons . r 
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prior earnings were the best single predictor of future .aployment and 
earnings.' Likewise, status as an AFDC applicant or recipient and being on 
welfare for more than two years were the best predictors of future welfare 
receipt. It is important to note that applicant/recipient differences in 
AFDC receipt are only partly explained by differences between the two 
groups in prior employment and welfare history. A difference in welfare 
receipt of 16 percentage points between applicants and recipients remains 
even after controlling for all other differences in characteristics. This 
is in part because substantial proportions of applicants are never approved 
(and therefore never go on welfare) and partly because approved applicants 
tend to leave the system faster than longe "-term recipients. 

Job readiness and dependency are both related to demographic charac- 
teristics, as is to be expected, but not in the same way. This also can be 
seen in Table 2.5. Age, marital status and age cf yoiv.nest child are 

Q 

important determinants of welfare but not employment status The effect 
of a high school diploma is similar for both . Prior earnings is a much 
better predictor of employment status than welfare status. Being a 
recipient rather than an applicant is not re" ed to employment status, and 
prior welfare history is much less strongly related to employment status 
than it is to welfare status. 

The estimates of Table 2.5 can be used to assign to each sample member 
a predicted future employment or welfare receipt rate. These can serve as 
scores on separate scales of job readiness and dependency. 

Because combinations of lew recent earnings and long welfare history 
may yield significantly more information than either alone, several single- 
trait and combination subgroups were also analyzed. Table 2.6 displays the 



TABLE 2.6 



AFDC CONTROLS: NORMAL EMPLOYMENT AND WELFARE BEHAVIOR 
FOR COMBINATIONS OF V\, MAJOR SUBGROUPS 



Subgroup 


1 

Percent 
of 

Appl iconts 


1 

Percent 

of 

Recipients 


Averoge Quorterly 
Empsoyment Rote, 
Quorters 4-6 (%) 


Percent Reccing 
Any aFDC Poyment, 
Quorter 6 (%) 


First Tier 










Appl iconts With No 
Prior AFDC 


31.2 




37.0 


31.2 


Appl iconts With $3000 
or More Prior Eornings 


27.6 




60.3 


35.9 


Second Tier 










AppMcont Returnees 


68.8 




37.0 


46.4 


Appl iconts With Less Thon 
$300D Prior Eornings 


72.4 


— 


28.1 


43.8 


Applicont Returnees With 
Less Thon $3000 Prior 
Eornings 


49.3 




28.1 


49.4 


T^ird Tier 










AM Recipients 


— 


10D.0 


22.7 


72.4 


Recipients With More Thon 

Turn Ypnr^ nn APnT 

InU ICUI d Ull MrUL 




/Z. V 


1 ft L 

19.6 


76.4 


Recipients With No Prior 
Eornings 




75.8 


14.6 


75.7 


Recipients With No Prior 
Eornings ond More Thon 
Two Yeors on AFDC 




58.3 


12.7 


79.6 



SOURCE AND NDTES: See Toble 2.4. 
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subgroups that were found important in the subgroup analysis of impacts, 
organized into three categories, with some overlap among them: 
lirst tier 

Applicants with ao prior AFDC 

Applicants with $3,000 or more prior-year earnings 
Second tier 

Applicants who have been n AFDC before and have returned to 
the rolls (i.e., applicant returnees) 

Applicants with less than $3,000 in prior-year earnings 

Applicant returnees with less than $3,000 in prior-year 
earnings 

Third tier 

All recipients 

Recipients with more than two years on AFDC 
Recipients with no prior-year earnings 

Recipients with no prior-year earnings and more than two years 



These three tiers correspond loosely to decreasing levels of job readiness 
and increasing evels of dependency, even though, as can be seen frcra the 
cable, the two concepts do not yield exactly the same rankings. Subgroups 
within tiers overlap, and there is sane overlap between the first and 
second tiers. The ranking is useful, nevertheless, as a way to summarize 
how the effectiveness of the services used in the five programs included in 
the study varies across levels of dependency (see Chapter 4, Section III). 

V. P rogram Costs 

In developing welfare employment policy, program impacts on employ- 



on AFDC 
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raent. earnings, welfare receipt and oth. • outcomes must be weighed against 
program costs. Cost differences by subgroup were calculated for the San 
Diego and Baltimore projrams. This section briefly describes these cost 
differences and discusses implications for the overall subgroup analysis. 
A more detailed discussxon of costs, together with an assessment of the 
benefit-cost implications of the subgroup impact and cost differences 
(i.e., the costs of the program over and above the costs of araployraent- 
related activities undertaken by controls) is available from MDRC. 

Table 2.7 presents gross program costs, expressed on a per experi- 
mental basis, for the San Diego and Baltimore programs. The figures 
include the costs of serving nonparticipants as well as participants in the 
experimental groups, and are broken down by major program component. They 
are also disaggregated for the two major subgroups based on prior earnings 
and welfare experience. 

Prior AFDC receipt was the most important characteristic associated 
with higher costs in both programs. The group with the longest welfare 
history, more than two years, had the highest costs. People in this sub- 
group stayed on welfare and in the programs longer and, in Baltimore, were 
assigned to the expensive services more often. Thus, for example, costs 
for recipients with more than a two-year welfare history were over 40 per- 
cent higher than for first-time applicants. In addition, in the Baltimore 
program, individuals without a high school diploma received more of the 
costly remedial education services than enrollees who already had a 
diploma. 

Thus, subgroup costs did vary and the expenditures were higher for the 
more dependent. This pattern was probably present in all five programs, 10 



TABLE 2.7 
SAN DIEGO AND BALTIMORE 
PROGRAM COSTS PER EXPERIMENTAL, BY PROGRAM AND MAJOR SUBGROUP 





' 1 

Totol Averoge 


Group Job 


Work 


Other Program 


Support 


Subgroup 


Cost 


Seorch 


Experience 


Activities 


Services 


Son Diego Appl iconts 


$786 


$560 


$91 


$96 


$38 


Prior Yeor Eornings 












$3000 or More 


843 


627** 


81 


96 


40 


$1-2999 


733 


522 


81 


96 


35 


None 


775 


537 


103 


96 


39 


Hod Own AFDC Cose 












Never 


729 


534 


66*** 


96 


33 


Two Yeors or Less 


794 


567 


91 


96 


40 


More Thon Two Yeors 


845 


585 


124 


96 


41 


Bol timore Appl iconts 0 


843 


173 


51 


329 


195 


Prior Yeor Eornings 












$3000 or Mc^e 


702* 


134 


nit 
37* 


294 


1 50* 


$1-2999 


949 


204 


50 


376 


216 


None 


879 


183 


63 


323 


214 


Hoo Own AFDC Cose 












Never 


804*** 


166 


28*** 


342*** 


111 * 

177*** 


Yeors or Less 


694 


146 


48 


251 


159 


More Thon Two Yeors 


1037 


209 


70 


408 


247 


loltimore Recipients 0 


1065 


188 


89 


386 


288 


Prior Yeor Eornings 












$3000 or More 


831 


192 


55 


344 


159 


$1-2999 


1041 


215 


82 


363 


267 


None 


1088 


180 


93 


396 


303 


Hod Own AFDC Ccse 












Never 


635*** 


126 


54* 


213* 


160** 


Two Yeors ot Less 


862 


163 


70 


315 


214 


More Thon Two Yeors 


1156 


200 


97 


420 


32 



SOURCE: MDRC colcuiotions from program cost ond enrollment doto (see Long ond Caspar, 1967). 

NUTcS: Estimotes ore totol costs Incurred for experimentoi s ond ore averaged over participants ond 
non-participants. F-tests were performed on vorlotlons in cost In eoch column for each subgroup dimension. 
Stotisticol signlf Iconce levels ore Indicoted os: * = 10 percent; ** * 5 percent; *** - 1 percent. 



°The cosl components listed for Boltlmore do not Include tfr" r osts of sanctioning, and thus do 
& not sum to totol cost. 
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although it was most pronounced in Baltimore. Chapter 4 ex ~ines whether 
these greater expenditures produced greater impacts on the more dependent 
subgroups. 

As that chapter makes clear, the overall subgroup variation in cost 
was small compared to the variation in impacts among subgroups. This was 
particularly true in San Diego, which had the same relatively short 
treatiuwnt sequence for all enrollees, and which did not include education 
and training. Decisions about alternative targeting strategies, there* re, 
hinge primarily on subgroup impact differences rather ti than cost 
differences. 

Before the discussion of subgroup impacts, Chapter 3 presents a brief 
discussion of the methodology used for the analysis. 
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CHAPTER 3 

MEASURING SUBGROUP IMPACTS AND ASSESSING PERFORMANCE INDICATORS 

This chapter reviews the principal elements of the experimental 
research design and the methodology used in this study. The discussion is 
meant as a general guide, although parts of it are inevitably somewhat 
technical. 

I. Experimental Design 

Any analysis of program impacts is based on a comparison between the 
observed outcomes of a program and what would have occurred without it. As 
explained in Chapter 1, program outcomes are relatively easy to observe. 
But estimating the program impact requires calculation of the difference 
between observed outcomes and what outcomes would have been in the absence 
of the program. 

A classical experimental design is often the preferred way of obtain- 
ing the standard for comparison. In such designs, clients are assigned on 
a random basis to either the experimental s-oup , members of which are 
eligible for program services, or to the control gro up, members of which 
are only eligible for the services available without the program. The 
average outcomes of experimental eligible for the program minus the 
average outcomes of controls are the program impact estimates. These 
measure the program achievements over and above the normal job-finding and 
welfare patterns of the eligible population. Random assignment ensures 
that the two groups are the same in terms of measured and, more important, 
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unmeasured characteristics, permitting unbiased estimates of program 
impacts. 

To maintain this lack of bias in the impact estimates, no changes can 
be made in the research group designations after random assignment, Experi- 
mentals" remain experimentals and ■controls' remain controls. Therefore, 
experimentals who did not, for some reason, participate in the progrars are 
still counted as part of the experimental group in the calculation of 
impacts. Program impacts, therefore, are expressed on a per experimental 
rather than a per participant basis. The definition of subgroups adheres 
to the same principle. Subgroups are defined by pre-existing character- 
istics observed at enrollment, not by any subsequent behavior or activity. 

II. Data Sources 

Earnings and welfare data were assembled from administrative records. 
The use of such records offers several advantages. First, administrative 
records can be much leos expensive than survey data, in part because 
registrants do not have to be recont acted during the follow-up. Records 
may also be more accurate than survey data because they do not depend on 
client recall of dollar amourts of earnings or welfare payments. Different 
rates of response by the experimental versus the control group — often a 
source of bias In survey data — are also not axpected with records data. 

Administrative records ai.e, nowever, limited in their comprehensive- 
ness and coverage. For example, quarterly earnings information can be 
obtained from the Unemployment Insurance (UI) system, but data on weeks 
worked, wages and hours worked are not available. Moreover, ths 
information can only be obtained w<th a lag, and sane delinquency in filing 
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earnings reports on 'he part of employers is common in wage-reporting 
states. Another drawback is that state UI systems do not normally record 
the earnings of people who commute to work across state lines or uncovered 
employment. Given random assignment, however, none of these factors should 
affect experimental and control, group outcomes differently. 

In addition, administrative records in this study contain no inform- 
ation on people other than the research sample members. They do not, for 
example, provide the earnings of other family members, A hose income (both 
earned and unearned) will affect a household's welfare dependency and 
general woll-being. 

The completeness and accuracy of the records data collected in this 
study were examined by comparing a small sample of data from the analysis 
tapes to the original paper or microfilm documents in state or county 
offices. Earnings and welfare payments were well matched. Further, a 
comparison of records and survey data from the Louisville WIN Laboratory 
and an earlier San Diego study suggests that the two sources yielded 
relatively similar information although the survey self -reports somewhat 
understated welfare receipt. 1 " 

Records data were merged with demographic and program activity 
information to form a single program data base, with a new record compiled 
for each sample member. Each record contains the client's employment 
background and welfare history in addition to aK series of outcome measures 
(quarterly UI earnings, monthly AFDC payments) running from the point of 
entry into the sample (i.u., the date of randan assignment) through to the 
end of the followup. Program activities and dates are also included. The 
earlier a person entered the sample, the more follow-up data are available. 
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No sample member has less than six quarters of earnings data and 18 months 
of welfare data. This is the minimum available in San Diego and Cook 
County. At least ten quarters are available in Baltimore and Virginia, 
twelve in Arkansas, 

The major data sources for all the programs analysed are summarized 
below: 

Client Information Sheets , one-page questionnaires filled out 
by client and staff as part of the randem assignment process, 
provide information on the demographic characteristics of 
sample members. All principal subgroups, with the exception 
of the subgroups identified by prior earnings, were defined 
using this information. 

State Unempl oy ment insurance (UI ) Earnings Records provide 
quarterly employment and earnings data reported by employers 
for each calendar quart?-: e.g., January, February and March; 
April, May and June. 

AFDC records supply information on monthly AFDC (i.e., 
welfare) grants. Monthly AFDC data are grouped by three-month 
periods, where the first month of the first quarter of 
follow-up is the luonth of enrollment. 

Unemployment insurance Benefit Records supply information on 
monthly UI benefit payments. 

Program Activity records provide information on program 
services, participation and deregistration. 

It is impoi -nt to note that Client Information Sheets were not available 
for the full impact sample in Cook County. For this reason, fewer sub- 
groups could be defined there, and those ore typically not directly 
comparable to the subgroups for the other four programs. 

Since randem assignwent can occur in the first, second, or third month 
of a calendar quarter, the first quarter of UI earnings can contain pre- 
program earnings for seme sample members. The first quarter of earnings is 
therefore not considered a follow-up quarter in the impact analysis and is 
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emitted from cumulative estimates of program impacts. 



III. Choice of Follow-up Period 

MDRC's research to date has shown certain patterns of outcomes for 
experimentals and controls over time. Typically, the outcomes for 
experimentals and controls were similar in the quarter of random assignment 
but began to differ in quarter 2, even though many experimentals did not 
join activities for as long as six months after enrollment. The experi- 
mental-control differences grew slcwly, with the difference often peaking 
at the one-year point or beyond. 

This report divides follow-up into an immediate post-random assignment 
period (quarters 1 through 3) and a longer-term follow-up period (quarter? 
4 and following). Quarters w re averaged — which helps to eliminate some 
of the transitory quarter-to-quarter variation in earnings. Earnings, as 
well as employment, AFDC receipt and payment amounts, are expressed as 
quarterly averages per person. Averages for the immediate and lon^^r-term 
outcomes were calculated separately. It should be emphasized that the 
longer-term averages contain more quarters of data for persons who entered 
the samples early. This averaging procedure has the disadvantage that it 
does not explicitly estimate juarter-by-quarter time trends in impacts. 

The longer-term follow-up period was selected as the focus of this 
subgroup analysis because subgroup differences appearing in the later 
quarters are the best indicators of long-run effects and are therefore 
likely to be more indicative of the total impact differences among 
subgroups. The training activities and education programs in Baltimore, 
which take as long as one year to complete, require a long follow-up 
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period, making it even more important to focus on the later periods. 

Statistical tests of significa are reported for differences between 
experimental and controls within subgroups. The differences between 
impacts for pairs of subgroups * 2re also tested; the results of these 
tests, which were not often statistically significant, are emitted frnm the 
tables but are occasionally mentioned wher* appropriate. Some of fcftese 
tests, along with other special statistical considerations relevant to the 
empirical comparisons among subgroups, are discussed in Appendix A. 

IV. The Subgroup Impact Regression Model 

A simple difference between average outcomes for experimental and 
control groups is sufficient to estimate impacts reliably in a carefully 
implemented experimental design. Use of linear regression lends extra 
precision to the estimates and corrects for minor differences in pre- 
program characteristics between experimental and controls. For this 
reason, the estimates reported in this paper are regress:ion-adjusted« 

In addition, regression techniques have been u^ad to produce two sets 
of subgroup impacts. The first set takes the point of view of the program 
administrator tfho asks: "Can I improve efficiency by targeting services to 
registrants with a single subgroup characteristic?' For example, it may be 
useful to find out if sample members with a high school diploma have 
different impacts than those without diplomas, ignoring differences in any 
other demographic characteristics (the kind of estimates shown in Chapter 
2, Table 2.4). These impact estimates arq unconditional estimates, and are 
the focus of Chapter 4. Such subgroup estimates do not take into account 
impact differences associated with other demographic and background 
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characteristics* For example, women without a high school dinloma 
generally have a weaker work record, but unconditional estimates do noc 
explain what part of the diploma effect is due to the work history 
characteristic itself rather than other characteristics of individuals with 
weak work records* Regres^.on, in th ' case, only serves the purpose of 
increasing precision and adjusting for minor pre-existing experimental- 
control differences* 

Tv;o or more characteristics can be included in unconditional 
estimation as interactions / and these are often useful to program 
operators* Tc continue the example above, the sample may be split four 
bays: person3 with and without diploma, further divided by employed/not 
employed in the recent pre-program period* Impacts calculated for each of 
these four subgroups may help to establish whether it is worthwhile to 
target services to a narrow subgroup defined by diploma and prior 
employment status. This approach provides information about targeting on 
the basis of two subgroup characteristics, without controlling for other 
factors* 

Regression an* lysis can be used to generate conditional estimates* 

These estimates held all subgroup haracteristics constant except the one 

in question* That is, any conditional impact difference associated with a 

high school diploma would indicate the importance of the schooling 

credential itself, eliminating effects due to prior employment record and 

other characteristics. If conditioning on prior employment status 

nullifiec the diploma effect, then the prior-employment difference across 

3 

diploma subgroups would be the "real" i. ason for the diploma impact. 

Both unconditional and conditional estimates are important, depending 
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on the questions asked. Unconditional estimates arc- presented and dis- 
cussed in the next chapter because they address questions of targeting with 
limited information. Conditional estimates, however, are required for the 
testing of performance measures in Chapter 5. Conditional estimates will 
b$- discussed in Chapter 4 only insofar as they raise issues regarding the 
conclusions drawn from the unconditional estimates. 

V. Testing Performance Indicators 

A handful of prior studies hive attempted to test the correlation 
between various measures of performance and net program impact. These 
studies generally did not have experimental corpar 3son data, but their 
techniques are similar to the ones used in this study of performance 
measures. 

The basic approach is as follows: 

3. Obtain an estimate of net program impact for each individual 
in the treatment group; 

2. Create a measure of program performance — e.g., did the 
sample member enter employment, what were his/her wages? 

3. Compute correlation coefficients between the net impact and 
the performance measures, with measures with the greatest 
correlation being identified as the "best* performance 
indicators; 

4. As a supplemental analysis, determine whether two indicators 
work better than one. Compute a regression of net impact on 
two performance indicators and report the coefficients and 
their statistical significance* In this way, it may be 
possible to determine that one indicator has more power than 
another or is a useful supp.lement. 

This procedure has remained approximately the same since studies in the 

mid-1970s correlated performance measures with the impacts of certain 

4 

pre-CETA employment programs. 
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The difficult part of this process is the first step: the estimation 
of a net impact for each individual. 5 

Early studies of performance indicators estimated individual-level 
impacts without experimental data, and thus had to depend on impact 
estimates from participant/nonparticipant comparisons adjusted by 
regression for various demographic and participation variables, *,uch as 
type of treatment and length of stay. Thus, while these studies have used 
essentially the same procedure to estimate individual impacts as used in 
the random assignment evaluations, the estimates they generated will be 
biased to the extent that the regression models used were not able to 
control adequately for differences between the participant and 
nonparticipant groups* 

Interpretation cf the correlations can be problematic, because they 
apply to the programs as implemented when the reason for testing 
performance indicators may be to establish standards that will change the 
way that programs are implemented* For this reason it is important, in 
assessing the validity of any particular class of indicators, to consider 
other evidence as well. 
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SUBGROUP DIFFERENCES IN IMPACTS 

This chapter sunraarizes program impact differences for subgroups of 
the WIN-mandatory AFDC caseload in the five work/welfare areas included in 
this report — San Diego, Baltimore, several counties in Virginia, part of 
Little Rock and one less urbanized county in Arkansas, and Cook County in 
Illinois. The main focus is on the subgroups of WIN-mandatory registrants 
listed in Chapter 2, defined according to prior welfare and employment 
experience. Defining the subgroups along the straightforward dimensions of 
prior welfare and earnings histrry makes the conclusions on targeting of 
direct use to progrcun operators, since the subgroups can be readily 
identified for a variety of eligible populations. 

One thrust of the findings is that the leas, dependent subgroups 
(e.g., those with prior-year earnings above $3,000 or no previous welfare 
receipt) generally experienced below-average program impacts, that were 
rarely statistically significant and often the smallest estimates of any 
major subgroup. These findings suggest that a policy of targeting low-cost 
program components only to those in the WIN-mandatoiy caseload who are most 
"job ready" is not efficient. Although such a policy may result in a high 
■entered employment" r~te among participants or a high rate of AFDC case 
closures, administrators cannot be confident that such an approach will 
lead to real impacts on earnings or income. 

Above average and statistically significant earnings impacts occurred 
most consistently for a subgroup in the middle range of dependency 



-53- 



90 



(applicants who .xad already had at least one spell on AFDC ) . The earnings 
impacts on the most dependent subgroups (e.g., those with no prior-year 
earnings who are welfare recipients and have prior welfare histories) were 
typically below average, and did not occur consistently across programs, 
Th'.s suggests that focusing solely on Jiese groups may also not be the most 
effect *.ve strategy, at least in the context of the mandatory, mass parti- 
cipation, relatively low-cost programs that are the focus f this study. 

There were no impacts on welfare payments for the least dependent 
groups. There were modest impacts on welfare receipt for groups in the 
mid- and high-dependency range, often not statistically significant, 
however. 

Impact diff' rences among subgroups defined according to character- 
istics other than prior work and welfare generally were not consistent 
across programs. The best predictor among these other characteristic*, was 
marital status. The bulk of welfare savings in the four programs for which 
there are data on marital status came from the not-married and never- 
married subgroups. 

The tables report tests for the statistical significance of impacts 
for the individual subgroups. Significance tests across subgroups were 
also performed. Estimates of impacts for one subgroup may be described as 
■larger" or "smaller" than those of another, but without passing the 
statistical criteria these relationships lack the high level of certainty 
generally applied in social science research. Subgroups which appear to 
have lf^ger impact estimates may be subjected to cross-subgroup statistical 
tests of confidence. Such tests were not usually statistically 
significant, but where they contributed to confidence in the estimated 




differences in impacts across subgroups they are noted. For example, 
earnings gains fcr applicant returnees were statistically significantly 
greater than those for new applicants and all recipients combined (i.e., 
the balance of the sample) for two of the three programs where such 
comparisons were possible. Selected tests of this sort, together with a 
discussion of other statistical issues involved in subgroup comparisons, 
appear in Appendix A. 

In each table in this chapter the subgroups are shown with the most 
employable and/or least dependent subgroup at the top and the least 
employable and/or most dependent at the bottom. It should be noted that 
not all subgroups could be defined for all program*. 

Ihis chapter is organized in three sections. The first section 
analyzes impacts for the major subgroups and focuses on cross-program 
comparisons. The second examines other results of interest within each of 
the programs studied. Since the patterns that emerge are not completely 
uniform across programs, across subgroups or across dependency measures, 
the discussion is inevitably somewhat complex. The final section of the 
chapter provides a sunmary of the major findings. 

I. The Major Subgroups 

One way to judge the magnitude of impact for a subgroup is to compare 
the estimates with those for another subgroup. Are they higher or lower? 
Is the subgroup estimate larger or smaller than the full sample estimate? 
Another straightforward way is to classify impacts as falling above or 
below sane cutoff value. In this chapter, employment and welfare cutoffs 
were selected so that impact estimates for about half the subgroups would 
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fall above and half below the cutoff values. Impacts estimated to be above 
the cutoff values are called above average, and those below are labelled 
below average. The cutoffs chosen are, however, technically more akin to 
the concept of a median than a mean. The choice of cutoffs is to some 
degree arbitrary; they could be lower or higher depending on the stringency 
needed. 

The cutoff criteria define above average impacts for each outcome 

measure, respectively, as : 

o employment rate increases of more than 4 percentage points a 
quarter 

o earnings increases of more than $100 a quarter 

o welfare receipt reductions of more than 2 percentage points a 
quarter 

o welfare savings of more than $20 a quarter. 
Estimates that fell belcw the cutoff on any of these definitions were 
considered below average even if statistically significant. Estimates that 
were above the cutoff but not statistically sign<f ^ :ant were considered 
uncertain. Some degree of arbitrariness is inherent in these cutoffs, 
since typical earnings rates and AFDC payment scheduler, differ across 
states and localities. It should also be emphasized again that program 
impacts are only one factor in targeting; other factors are also -mportant, 
depending on program objectives. 

The major subgroups used in the analysis are defined along three 
dimensions: whether they were applicants or recipients when they entered 
the study sample, whether they worked and how much they earned in the year 
before sample entry, and whether and how long they had been on welfare at 
sample entry. 



The minimum length of fol.\ow-up for earnings and AFDC payment data is 
six quarters in San Diego and Cook County, ten quarters jn Baltimore and 
Virginia, and 12 quarters in Arkansas. 

The most important definitional problem for subgroups concerns the 
distinction between applicants and reci^ nts in Cook County. Since this 
information was not recorded explicitly for the research sample the 
distinction had to be approximated. Sample members w.io received no AFDC 
payment in the three months prior to random assignment were classified as 
applicants; all others were classified as recipients. This procedure would 
have yielded groups that were relatively comparable to the groups in other 
states except that, in Cook County, only applicants whose grants had 
already been approved were enrolled in the program. The applicant samples 
the other states included all applicants, whether their grants were 
^sequently approved or not. The significance of this for the research is 
that the Cook County applicant subgroups had higher welfare receipt du~ „ng 
the follow-up period than did the other applicant subgroups. 

A. Welfare Status 

The evidence suggests that employment and earnin 'f 4. us are larger 
for applicants than for recipients, although the evidence «iot strong and 
is not completely consistent across programs (Table 4.1). loyment and 

earnings impacts for applicants were larger than those fo: recipients in 
two of the four programs that served both categories, although the 
increment was statistically significant in only one sample. Estimates for 
the two groups were similar in a third program (Virginia). Employment 
increases were over 4 percentage points for four of the applicant samples 
but only for one recipient sample. At the same time, earnings gains topped 
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TABLE 4.1 



AFDC APPLICANTS AND RECIPIENTS: UNCONDITIONAL IMPACTS ON 
EMPLOYMENT, EARNINGS, AFDC INCIDENCE AND PAYMENTS, 
BY PROGRAM AND WELFARE STATUS 
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TABLE 4 J (continued) 



SOURCE: MDRC cilculotions from the County of Son Di<*go welfore records ond Mnempioymtnt Insuronce 
records from the EPP mformotion System; from the Stote of Morylond welfore ond Unemployment Insuronce 
records; from the Commonweolth of Virginio Unemployment Insuronce eornmps records, welfore records from the 
Virginio Automoted Client Information System, ond Foirfox County AFDC cose files; from tM Stote of Arkonscs 
welfore ond Unemployment Insuronce records; ond from the Illinois Deportment of Public Aid AAID system records 
•*nd the Illinois Unemployment Insuronce system eornings records. 

NOTES: These doto ore regression-od justed using ordinory icost squores. controlling for pre-rondor, 
assignment characteristics of somple members. Dol lor-denominuted 'motes include zero volues for sompU 
members not employed or for somple members not receiving welfore. c&rimotes tor oppiiconts ond recipients 
we # e obtoined from seporote regressions for eoch progrom. There mo be some ducreponcies in colculoting suits 
ond differences due to rounding. 



Somple Six j ore os follows: 

Appl iconts Recipients 

Son Diego 2361 

Boltimore 1380 1377 

Virginia 1269 16f1 

Arkonsos 670 457 

Cook County 2668 5167 

Samples for Son Diego ond Illinois exclude o second c.r^nmentoi group, not onoiyzea in this report. 

A two-toiled West wos applied to differences belt n en experimentcl ond contrci group*. Statistical 
significance levels ore indicated as: * = 10 percent; ** = 5 percent, *** = 1 percent. 

°Percent of ful I somple. 

b Son Diego served only nppl iconts. 

C The definitions of 'opplicrn/ ond "recipient* for Cook County differ frair the other programs. 
See text for discussion. 
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$100 per quarter for three of the five applicant samples but none of the 
recipient samples. Averaging across programs (giving each program equal 
weight) yields earnings gains that were about twice as large for applicants 
as for recipients. 

A supplemental analysis using regression analysis to control for 
demographic differences across sample members indicated that the impact 
differences did indeed stem from the applicant/recipient distinction, and 
not from other factors. 1 

Ircpicts on welfare do not follow a clear subgroup pattern. Reductions 
in the percent receiving AFDC were over 2 percentage points for four of the 
five applicant samples versus only one of the four recipient samples, but 
typically were not statistically significant. The only statistically 
significant impacts were in Arkansas, where the reduction was larger for 
recipients than for applicants. Impacts on AFDC payments for applicants 
versus recipients show no clear pattern. 

B. Prior Earnings 

Among applicants, the prior earnings results indicate that the most 
employable groups did not have the largest impacts (see Table 4.2). Only 
one program (Virginia) achieved its maximum employment and earnings 
increases for applicants with $3,000 or more in prior-year earnings. Two 
programs (Baltimore and Arkansas) showed maximum impact for the subgroup 
with prior-year earnings in the $1 - $2,999 range. San Diego showed the 
largest gains among applicants with zero earnings in the prior year. 

Among recipients, since very few people had prior earnings in the top 
earnings category, all individuals with prior-year earnings were grouped 
together. For recipients with seme prior-year earnings only one of che 



TABLE 4.2 



AFDC APPLICANTS AND RECIPIENTS: UNCONDITIONAL IMPACTS ON EMPLOYMENT. EARNINGS, 
AFDC INCIDENCE AND PAYMENTS. BY PROGRAM, WELFARE STATUS. 
AND PRIOR YEAR EARNINGS SUBGROUP 



Subgroup, Wei fore 
Stotus, ond Progrorc 


i 

Percent 
of 

o 

Somple 


1 

Percent Employed Quorterly 
Quorters 4 - Lost (%} 


Averoge Eornings Per Quorter 
Quorters 4 - Lost ($) 


Experimental Control 


Difference 


Experimentol Control 


Difference 


Prior Yeor Earnings 


















Appl icont 


















$3000 or More 


















Son Diego 


28.8 


60. 7 


58.7 


+1 


.9 


1444 


1482 


- 39 


Baltimore 


3i - 9 


65.0 


62.5 


+2 


.5 


1453 


1435 


+ 16 


Vi rginic 


29.2 


65.6 


56.1 


+9 


.5** 


1348 


1041 


+307*** 


Arkonsos 


12.5 


60.8 


53.4 


+7 


.3 


896 


884 


T 1 L 


Cook County b 


33.0 


53.9 


52.2 


+1 


.7 


1373 


1336 


+ 0/ 


$1-2999 


















Son Diego 


22.9 


42.9 


41.7 


+ 1 


.2 


813 


729 


+ 84 


Baltimore 


29.3 


51.5 


45.0 


+6 


.5* 


1 068 


799 

/ Li 


+339*** 


Virginia 


28.4 


52.4 


48.2 


+4 


.2 


801 


716 


+ 83 


Arkonsos 

« _ h 


22.5 


56.9 


43.3 


+13 


.6*** 


795 


598 


+ 17/* 


Cook County 


20.4 


33.5 


38.4 


-4 


.8 


555 


722 


-167 

1 u / 


None 


















Son Diego 


48.4 


30.3 


22.8 


+7 




601 


375 


+225*** 


Bol timore 


38.8 


27.7 


23.6 


+4 


1 


569 


396 


+ 171* 


Vi rginio 


42.3 


31.7 


30.9 


+0 


8 


4 65 


544 


- 79 


Arkonsos 
Cook County^ 


64.9 


15.8 


10.6 


+5 


3* 


240 


146 


+ 95 




15.0 


16.5 


-1 


5 


268 


301 


- 33 


Recipients 


















Some 


















Baltimore 


27.2 


50.3 


52.7 


-2. 


4 


799 


970 


-171** 


Vi rginio 


23.0 


51.8 


47.2 


+4. 


5 


848 


783 


+ 65 


Arkonsos 


8.5 


3*. 4 


46.3 


-6. 


9 


4 86 


578 


- 92 


Cook County b 


19.8 


42.2 


38.0 


+4, 


2*« 


865 


685 


+179*** 


None 


















Bol timore 


72.8 


23.9 


19.1 


+4. 


7** 


343 


240 


+ 104** 


Virginio 


77.0 


24.7 


20.8 


+3. 


9** 


363 


293 


+ 70 


Arkonsos 


91.5 


10.6 


7.2 


+3. 


4 


118 


89 


+ 29 


Cook County b 


80.2 


14.4 


13.0 


+ 1. 


3 


223 


211 


+ 12 



(continued) 



TABLE 4.2 (continued) 



T 



Subgroup, Wei fore 
Storus, ond Progrom 


Do rroni 

nf 

0 


Percent Receiving AFDC Monthly 
Quorters 4 - Lost {%) 


Averoge AFDC Poyments Per Quarter 
Quorters ^ - Lost (S) 


Experimental Control 


Difference 


Experimental Control 


Difference 


Prior Yeor Eornings 


















Appl iconts 


















$30G0 or More 


















Son Oiego 


28.8 


26.5 


75 9 




A 
. o 


T7A 


OZO 


+ 3 


Bol 1 imore 


31.9 


36.5 


34.8 


+1 


.7 


295 


288 


+ 7 


Vi rg i ni o 


29.2 


22.7 


21.9 


+c 


.6 


182 


174 


+ 8 


Arkonsos 


12.5 


27.2 


26.7 


+0 


.5 


138 


137 


+ 1 


Cook Count y b 


33.0 


46.4 


47.0 


-0 


.6 


406 


403 


+ 3 


$1 -2999 


















Son Diego 


77 Q 


Tfi 7 

<J Urn I 


7 


L 


0 

. 7 


a no 


^71 


-63 


Bol t imore 


29.3 


42.2 


46.0 


-5 


.8 


367 


4 05 


-38 


Vi rginio 


28.4 


24.8 


28.2 


-3 


.4 


201 


240 


-39 


Arkonsos 


77 5 


25.2 


33.6 


-8 


.4 


126 


177 


-51* 


Cook County b 


20.4 


56.7 


59.3 


-2 


.6 


460 


493 


-33 


None 


















Son Dieoo 




36.5 


38.9 


-2 


.4 


514 


554 


-40 


Bol t imore 


TP R 


48.8 


51.9 


-3 


.2 


424 


437 


1 O 


Vi rg in i o 


A? 7 

*T i. > O 


23.3 


27.8 


-4 


.5 


189 


215 


-26 


Arkonsos 


64.9 


24.4 


28.9 


-4 


.4 


130 


152 


-22 


Cook County 


46.5 


59.4 


62.8 


-3 


.4 


519 


540 


-21 


Recipients 


















Some 


















Bolt imore 


27.2 


62.5 


58.8 


+3 


.7 


528 


502 


+26 


Virginia 


23.0 


41.1 


41.2 


-0 


.1 


326 


342 


-16 


Arkonsos 


8.5 


43.0 


46.1 


-3 


1 


206 


241 


-35 


Cook Coun?y b 


19.8 


62.1 


67.0 


-4 


8** 


567 


607 


-40* 


None 


















Bolt imore 


72.8 


73.1 


74.4 


-1 


2 


666 


666 


+ 1 


Virginio 


77.0 


54.2 


55.3 


-1 


1 


437 


464 


-26 


Arkonsos 


91.5 


54.0 


65.2 


-11 


.3*** 


291 


353 


-63*** 


Cook County 


80.2 


79.2 


79.9 


-0 


6 


772 


778 


- 6 



SOURCE AND NOTES: See Toble 4.1. 
o 



Percent of appl Iconts and percent of recipients. 
b„ 



The definitions of •applicant" and •recipient* for Cook County differ from the 
rri ^^her programs. See text for discusion. 



four programs had above average effects on earnings. The great majority of 
recipients had no prior-year earnings. Only one program had an above 
average effect on this group although all the impacts were positive. 

Welfare effects were again smaller than employment and earnings 
effects. It is of considerable interest to note that applicants with the 
best prior earnings records had no dollar welfare savings in any of the 
five programs. This was true even in Virginia, where this subgroup 
obtained above-average earnings increases. In contrast, all ten of the 
welfare receipt estimates for applicants with $1-2,999 or zero prior 
earnings ( in the middle of the employability range ) showed reductions 
greater than 2 percentage points, although none was statistically 
significant. Ml except one of these subgroups also showed above average 
dollar savings, though again not generally statistically significant. Only 
three of the reductions among recipients exceeded 2 percentage points, 
although seme recipient groups did have dollar savings. 

C. Prior Welfare 

Impacts by length of prior welfare receipt are shown in Table 4.3. 
The Cook County sample is excluded, since information about length of prior 
welfare was not available. 

With respect to employment and earnings, the subgroups at the top of 
the breakdown shew no consistent increases. These first-time applicants 
are clearly the least dependent as a group, although seme of them may have 
received welfare on their mother's grant as a child. Nor are any employ- 
ment or earnings effects statistically significant for this subgroup. Only 
one employment estimate (Arkansas) is greater than a 4 percentage point 
gain; only one earnings impact (Baltimore) exceeds $100. 
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TABLE 4.3 



AFDC APPLICANTS AND RECIPIENTS: UNCONDITIONAL IMPACTS ON EMPLOYMENT, 
EARNINGS, AFDC INCIDENCE AND PAYMENTS, BY PROGRAM, 
WELFARE STATUS, AND AFDC HISTORY SUBGROUPS 



Subgroup, Wei fore 
Stotus, ond Program 


Percent 
of 

. t 0 

Somple 


I 

Percent Employed Quorterly 
Quorters 4 - Lost {%) 


1 

Average Eornings Per Quarter 

Quorters 4 - Lost (*) 


Experimental 


Control 


Difference 


Experimental 


Control 


Difference 


Hod Own AFDC Cose 

















Appl iconts 
















Never 
















Son Diego 


33.4 


44.1 


41.8 


+2.3 


1018 


981 


+ 37 


Bo J t irr.ore 


22.7 


46.6 


47.1 


-0.4 


1136 


1015 


+121 


Vi rginic 


26.2 


44.9 


43.0 


+ 1.9 


868 


881 


- 13 


Arkonsos 


56.3 


27.3 


22.4 


+4.9 


394 


368 


+ 26 


Two Yeors or Less 
















Son Diego 


38.7 


41.4 


35.3 


+6.1** 


898 


732 


+165** 


Bol Timore 


41.8 


51.7 


45.9 


+5.8** 


1109 


940 


+ 169* 


Virginio 


31.7 


50.1 


43.2 


+7.0* 


886 


719 


+169* 


Arkonscs 


36.7 


35.8 


25.6 


+10.2** 


541 


329 


+212** 


More Thon Two Yeors 
















San Diego 


27.9 


40.0 


35.2 


+4.8 


731 


585 


+ 146 


Bol timore 


35.5 


40.6 


35.0 


+5.6* 


776 


563 


+208** 


Virginio 


42.2 


47.2 


43.2 


+3.9 


736 


663 


+ 73 


Arkonsos 


7.0 


31.9 


18.6 


+13.3 


402 


194 


+208 


Recipients 
















Two Yeors or Less b 
















Bol timore 


26.2 


43.0 


39.0 


+4.0 


753 


638 


+115 


Virginio 


28.2 


34.8 


35.0 


-0.2 


577 


595 


- 18 


Arkonsos 


35.0 


19.6 


16.9 


+2.7 


241 


214 


+ 27 


More Thon Two Yeors 
















Bol timore 


73.8 


26.9 


24.5 


+2.4 


367 


368 


- 0 


Virginio 


71.8 


29.5 


23.6 


+5.8*** 


436 


327 


+110** 


Arkonsos 


65.0 


9.6 


7.1 


+2.4 


100 


85 


+ 14 



(continued) 
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TABLE 4.3 (continued) 



^uhnrfiun wp! f rxrc. 

ouuyi \J\jyi |fci I Uf c 

jiu lui, unu rroarum 


T 

Percent 

of 
- o 
Sompl e 


1 

Percent Receiving AFDC Monthly 
Quarters 4 - Lost {%) 


1 

Averoge AFDC Poyments Per Quarter 
Quorters 4 - Lost ($) 


Experimental Control 


Difference 


Experimental Control 


1") t 4 4 or Arc 


Hod Own AFDC Cose 
















Appl i conts 
















Ngy e r 
















Son nipfio 


00 • *i 


22.7 


23.0 


-0.2 


314 


320 


r 

- 5 


Bolt i m<ir£ 


11. J 


33.3 


35.8 




287 


295 


- 9 


Vi r9inio 


26.2 


16.1 


20.0 


-3.8 


132 


160 


-28 


Arkansas 


56.3 


1 0 1 
1 7 • 1 


Z4*Z 


-5.1 


96 


127 


-31 


Two Years or Less 
















Son Diego 


OR 7 

0 0./ 


33.1 


36.9 


_*3 7 
O • / 


436 


510 


-74* 


OUI II Hi Ul c 


41.0 


40.2 


40.9 


-0.7 


344 


352 


- R 


Virginia 


31.7 


20.3 


23.7 


-3.4 


169 


1 ftp 


1 7 


Arkonsos 


36.7 


33.7 


35.7 


-2.0 


181 


189 


- 8 


Unr P Thnn Tun Vo nr c 
















ou ii v } c y U 


27 .9 


42.6 


43.0 


-0.4 


580 


589 


- 8 


Bol timore 


35.5 


52.3 


56.7 


-4.4 


443 


466 


-25 


Vi rgini o 


42.2 


30.5 


31.8 


-1.3 


243 


257 


-14 


Arkonsos 


7.0 


23.9 


39.5 


-15.6 


129 


204 


-75 


Recipients 
















Two Yeors or Less b 
















Bol timore 


26.2 


54.8 


58.7 


-3.9 


483 


508 


-25 


Vi rgini a 


28.2 


41 .0 


37.7 


+3.3* 


324 


305 


+19 


Arkonsos 


35.0 


37.5 


54.4 


-16.9*** 


201 


293 


-92** 


More Thon Two Yeors 
















Bol timore 


73.8 


75.7 


74.2 


+1.5 


680 


661 


+19 


Vi rgini o 


71.8 


5<.8 


58.6 


-3.8* 


444 


492 


-48** 


Arkonsos 


65.0 


61.7 


69.0 


-7.3 


329 


373 


-44* 



SOURCE AND NOTES: See Toble 4.1. 



Percent of oppli conts ond percent of recipients. 



Includes o smol I number of recipients who reported never having hod their own AFDC 

cose. 
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Further down the table, applicants with a welfare history had employ- 
ment and earnings impacts that were almost always above average and were 
more often than not statistically significant. Among these applicant 
returnees, the length of prior welfare history did not make much differ- 
ence, although the impacts for those with two years or less were more 
likely to be statistically significant. Seven of the eight employment 
impacts for these two subgroups were over 4 percentage points; seven of the 
earnings impacts were over $100. If estimates were available for Cook 
County, they would probably not show impacts for these subgroups, since 
neither applicants as a whole nor any of the applicant subgroups that can 
be broken out showed much impact there. With respect to welfare savings, 
the only clear pattern was that in no program did first-time applicants — 
the least dependent subgroups — have the largest impacts. 

D. Combination Subgroups 

As shown in Chapter 2, the combination of low prior earnings and a 
long welfare history leads to longer periods of time on welfare than does 
either of those characteristics alone* It is therefore of interest to 
examine program impacts for different earnings/welfare history 
combinations. 

Table 4.4 presents impact results for several pairs of such subgroups. 

E£Ch pair consists of a less dependent and a more dependent subgroup. The 

combinations were chosen to split each of the applicant and recipient 

2 

samples into two parts as equal in size as possible. The more dependent 
of the two applicant subgroups was defined as follows: 
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TABLE 4.4 



AFDC APR. I CANTS AND RECIPIENTS: IMPACTS FOR SUBGROUPS 
COMBINING PRIOR EARNINGS, PRIOR AFDC RECEIPT, AND HIGH SCHOOL DIPLOMA STATUS 





T 




Eornings Impoct, Ouorlers 4 - Lost 


($) 






Appl iconts 


1 

Recipients 


Subgroup 


Son Diego 


Bol timore 


Virginio 


Arkonsos 


Bol timore 


Virginio 


Arkonsos 


Lcwer Prior Eornings 
Plus Higher Prior AFDC 0 

No 

Yes 


+ 87 
+151** 


+ 86 
+253*** 


+ 153* 
+ 20 


+ 36 
+202** 


-48 

+88 


+35 
+94* 


- 0 
+28 


Lover Prior Eornings Pius 
Higher Prior AFDC Plus 
No High School Diplomo 

No 

Yes 


+109* 
+ 158 


+174** 
+ 165 


+ 75 
+ 95 


+ 95* 
+ 150 


+ 14 
+57 


+63 
+78 


+ 0 
+43 





f 


AFDC 


Poyment Impact, Quorters 4 - Lost 


(1) 






Appl iconts 


1 

Reci pients 


Subgroup 


Son Diego 


Bol timore 


Virginic 


Arkonsos 


Bol timore 


Virginio 


Arkonsos 


loner Prior Eornings 
Plus Higher Prior AFDC 0 

No 

Yes 


- 3 
-63* 


- 8 
-19 


- 9 
-29 


-29 
-22 


+l£ 
- 1 


+ 8 
-48** 


-83** 
-48* 


Lcwer Prior Eornings Plus 
Higher Prior AFDC Plus 
No High School Diplomo 

No 

Yes 


-26 
-69 


-11 
-19 


- 2 
-63* 


-16 
-54* 


+20 
-14 


+ 4 

-69*** 


-69** 
-48 



SOURCE AND NOTES: See Toble 4.1. 



'La*er prior eornings' is defined for applicants os eornings of less thon $3000 in the 
yeor prior to rondom ossignment; for recipients it is zero eornings. 'Higher Prior AFDC nieons any prior AFDC 
for oppl iconts ond more thon two yeors for recipients. The regression model utilized differs from thot 
employed previously by the introduction of o Interoctlon term for the subgroup combination. 



ERLC 



Prior-Year Earnings 



None 



$1-2,999 



$3,000 or More 



Had Own AFDC Case 



Never 



no 



no 



no 



Two Years or Less 



yes 



yes 



no 



More Than Two Years 



yes 



yes 



no 



The more dependent subgroup of applicants, therefore, contains returnees 
with prior-year earnings of less than $3,000. The less dependent subgroup 
of applicants includes those applicants with no welfare history and those 
applicant returnees with high prior-year earnings, 

A similar split was made for recipients: The less dependent subgroup 
contains all individuals with any earnings or with prior welfare experi- 
ence of two years or less. The more dependent subgroup contains those with 
no earnings and more than two years of prior welfare experience. 

Two additional pairs of subgroups were created by moving individuals 
with a high school diploma from the more dependent groups to the less 
dependent groups in both the applicant and recipient categories; the more 
dependent recipients in this categorization are the most dependent 
subgroups of all. In Table 4.4, the more dependent subgroup of each pair 
is indicated with a 'yes" label and is displayed as the lower row. The 
Cook County sample was excluded because there was no information on length 
of prior welfare history. 

The results follow closely from those already discussed. With respect 
to earnings impacts among applicant subgroups, the results for the top pair 
of applicant subgroups indicate that the less dependent have below average 
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impacts in three out of four programs, and the more dependent have above 
average impacts. Adding education to the combination of characteristics to 
increase the dependency contrast indicates, howevei, that after a certain 
point on the dependency scale, the impacts become smaller again. The 
implication is that earnings impacts per enrollee at first tend to increase 
as dependency increases, but eventually stabilize or begin to decline as 
the most dependent end of the spectrum is approached. 

Among recipients, earnings impacts were somewhat larger for the more 
dependent half of each pair, but the differences were not as large for 
applicants. The impacts on AFDC payments were typically not statistically 
significant, although the pattern for applicants suggests that the impacts 
may be greater for the more dependent. 

II. Further Discussion of Subgroup Differences Across Programs 

In comparisons across programs, characteristics other than prior 
earnings and welfare receipt — such as education and numbers of children 
-^"usually produced conflicting relationships across program samples. They 
were, however, sometimes helpful in interpreting interactions between 
individual characteristics and specific features of each program. 

Table 4.5 presents impacts on earnings and welfare payments for appli- 
cants and recipients in the five programs under discussion, according to 
education, numbers and ages of children, marital status, and other charac- 
teristics. As before, the impact estimates sta>rt at the fourth quarter 
after random assignment and average all quarters through to the end of the 
observation period for each program. The few differences between immediate 
and longer-term impacts that are important are pointed out where relevant. 
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TABLE 4.5 



AFDC APFtlCANTS AND RECIPIENTS: UNCONDITIONAL IMPACTS ON EARNINGS AND 
AFDC PAYMENTS, BY PROGRAM, MINOR SUBGROUP, AND WELFARE STATUS 



Subgroup, Wei fore 
Stotus, ond Progrom 


1 

Percent 
of 

SamDl e^ 

v\Jlllh/| v 


1 

Average Eornings Per Quorter 
Quorters 4 - Lost (%) 


Averoge AFDC Poyments Per Quorter 
Quorters 4 - Lost ($) 


Experimental 


Control 


Difference 


Experimental 


Control 


Difference 


High School DiplOQO 
















AodI 1 conls 
















Yes 
















Son Dleoo 


61.5 


1068 


923 


+146** 


375 


420 


- 44 


Bol timore 


44 ,9 


1199 


1106 


+ 92 


337 


337 


- 0 


Vlrglnio 


50.8 


939 


912 


+ 27 


175 


155 


+ 21 


Arkonsos 


55 . 1 


557 


397 


+161** 


118 


127 


- 9 


Cook County b 


46.8 


958 


957 


+ 1 


442 


457 


- 15 


No 
















Son Dieoo 


38.5 


60? 


534 


+ 74 


532 


547 


- 15 


Bol timore 


55,1 


829 


593 


+236*** 


391 


416 


- 25 


Vlrglnio 


49.2 


694 


559 


+ 136* 


206 


267 


- 61** 


Arkonsos 


44.9 


313 


270 


+ 43 


143 


190 


- 47** 


Cook County** 


53.2 


46 1 


532 


- 71 


495 


510 


- 16 


Rec I d! enls 

r\v ^ i w i will w 
















Yes 
















Bol timore 


42.1 


645 


59e 


+ 46 


557 


546 


+ 10 


Vlrglnio 


38.8 


650 


569 


+ 80 


390 


378 


+ 12 


Arkonsos 


42.0 


180 


212 


- 32 


245 


309 


- 64* 


b 

Cook LOunty 


30.9 


532 


467 


+ 66* 


663 


684 


- 20 


No 
















Bol timore 


57.9 


347 


3 17 


+ 30 


679 


677 


+ 2 


Vi rgini o 


61.2 


363 


302 


+ 61 


426 


474 


- 48** 


Arkonsos 


58.0 


129 


73 


+ 56 


311 


369 


- 58** 


Cook County^ 


69.1 


269 


230 


+ 39 


761 


771 


- 10 


Chi 1 d 12 or Under 
















Annl I rnnts 
















No 
















Son Diego 


22.6 


1001 


676 


+323*** 


293 


341 


- 48 


Bol tiroore 


27.5 


954 


942 


+ 12 


256 


244 


+ 12 


Vlrglnio 


22.9 


815 


636 


+ 180 


133 


160 


- 28 


Arkonsos 


9.6 


384 


206 


+1 78 


83 


115 


- 32 


Yes 
















San Dieoo 


77.4 


858 


803 


+ 55 


477 


506 


- 29 


Bol timore 


72.5 


1 01 1 


"f Oft 

780 


+232*** 


408 


432 


- 24 


Virgin'o 


77.1 


821 


771 


+ 49 


208 


224 


- 1 7 


Ar konsos 


90.4 


455 


356 


+ 100* 


135 


160 


- 25* 


Recipients 
















No 
















Bol timore 


13.4 


271 


242 


+ 28 


438 


460 


- 22 


Virginia 


23.7 


418 


355 


+ 63 


258 


286 


- 28 


Arkonsos 


11.8 


37 


58 


- 21 


171 


251 


- 79 


Yes 
















Bol timore 


86.6 


504 


46o 


+ 38 


657 


647 


+ 10 


Virginia 


76.3 


492 


421 


+ 70 


459 


482 


- 23 


Arkonsos 


88.2 


165 


141 


+ 24 


298 


356 


58»* 
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TABLE 4.5 (continued) 



Subgroup. We! fore 
Stotus. ond Progrorr 


Percent 

of 
Son pi e 


Average Eornings Per Quorter 
Quarters A - Lost {%) 


1 

Averoge AFDC Poyments Per Quorter 
Quorfers 4 - Lost {$) 


F v no r 1 >>p ntnl 


LUnT 1 UI 


ui t Terence 


Exper Ircentol 


Control 


Oifference 


Number of Own Children 
















Appi iconts 
















One 
















Son Diego 


49.7 


887 


80? 


+ 80 


346 


355 


- 9 


Bol timore 


50.4 


1033 


766 


~*. D / 


900 


^97 
Oil 


- 9fl 

— 1 0 


Virginia 


49.6 


781 


ftOA 

0 7 C 


4- R5 


1 7 A 
\ I H 


1 DC 

1 85 


i 1 
- 1 1 




42.5 


427 


391 


+ 36 


106 




- ?1 


More Tnon One 
















Son Diego 


50.3 


895 


740 


+155** 


595 


5Rfl 
J ou 


— JO 


Bol timore 


49.6 


966 


891 


+ 75 


All 




, ft 

+ u 


Virginio 


50.4 


856 


780 


+ 76 


207 


234 




Arkonsos 


57.5 


467 


305 


+161** 


146 


1 76 


- 9fl 


Reel pients 
















One 
















Bol timore 


43.1 


528 


493 


+ 35 


5m 


599 


- 1 0 


V i rni n i n 

* II VJ 1 1 1 1 u 


a*) n 

4 L • U 


462 


426 


+ 56 


^/5 
w£ J 


1l\ 


«. 1 7 


Arkonsos 


34.6 


125 


173 


- 46 


192 


257 


- A5* 


More Tnon One 
















Bol timore 


£6.9 


430 


392 


+ 36 


721 


696 


+ 94 


Virginio 


58.0 


469 


391 


+ 77 


■1 / w 




- 90 


Arkonsos 


65.2 


164 


109 


+ 56 


WW A 


j yu 


C P»» 

- io** 


Currently Morried 
















Appl Iconts 
















Yes 
















Son Diego 


46.6 


852 


750 


+ 102 


469 


450 


+ 19 


Bol timore 


50.4 


951 


829 


+ 122 


Ik? 

JO A. 


* 1A 
w M 


1 0 


Virginio 


49.3 


820 


667 


a.1 59» 

T 1 J L 


1 7 1 


1 7 A 


r 

5 


A rtnncn c 
HI KOlidlJ d 


J 1 . 7 


379 


274 


+ 105 


117 


159 


- ^5 


No 
















Son Diego 


53.4 


925 


794 


+ 131* 




4 R9 


— 7 All 

— / 0" 


Bol timore 


49.6 


1046 


824 


+222*** 


370 


387 


- 1 ft 


Virginic 


50.7 


819 


810 


+ 9 


210 


243 


- 33 


Arkonsos 


68.1 


481 


373 


*108* 


136 


158 


- 22 


Recipients 
















Yes 
















Bol timore 


34.3 


436 


422 


+ 14 


614 


617 


- 3 


Virginio 


38.3 


464 


427 


+ 37 


433 


422 


+ 11 


Arkonsos 


21.2 


173 


242 


- 69 


311 


293 


+ 16 


No 
















Bol timore 


65.7 


490 


442 


+ 48 


634 


624 


+ 10 


Virginio 


61.7 


481 


394 


+ 87* 


398 


442 


- 44** 


Arkonsos 


78.8 


145 


103 


+ 43 


274 


356 


- 82*** 



(continued) 



TABLE 4.5 (continued) 



Subgroup, Wei fore 
Stotus, ond Program 


Percent 

of 
Sompl e 


Averoge Earnings Per Quorter 
Quarters 4 - Last ($) 


Averoge AFDC Poyments Per Quorter 
Quorters 4 - Lost ($) 


Experimental 


Control 


Difference 


Exoerifnenta 


Control 

1 1 1 1 VI 


Di f f erpncp 


Ever Married 
















Appl i conts 
Ve. 
Son Diego 
















84.1 


908 


806 


+102* 


421 


445 


- 24 


Bol limore 


69.9 


1003 


821 


+182** 


346 


363 


- 17 


vi rgi nio 


74.2 


830 


705 


+125*- 


170 


176 


- 6 


Arkonsos 


56.9 


424 


310 


+1 14* 


112 


123 


- 11 


No 
















Son Diego 


15.9 


805 


608 


+ 196 


511 


591 


- 81 


Bo! limore 


30.1 


980 


832 


+ 148 


413 


421 


- 8 


Virginia 


25.8 


789 


841 


- 52 


249 


308 


- 59* 


Arkonsos 


4 3 . 1 


481 


383 


+ 98 


153 


199 


- 45** 


Recipients 
















Yes 
















cai limore 


49. 1 


458 


425 


+ 33 


621 


624 


- 2 


Virginia 


65.3 


462 


376 


+ 86* 


400 


4G2 


- 2 


Arkonsos 

No 

Bol timore 


41.8 


159 


158 


+ 1 


329 


328 


+ 1 


50.9 


487 


446 


+ 40 


633 


620 


+ 13 


Vi rginio 


34.7 


497 


459 


+ 38 


432 


495 


- 63** 


Arkonsos 


58.2 


143 


11 2 


+ 32 


249 


353 


-104*** 


Age 
















Appl i conts 
















30 or Over 
















Son Diego 


65.6 


966 


776 


+189*** 


411 


464 


- 54* 


Boi timore 


65.4 


1076 


927 


+150* 


349 


347 


+ 2 


Vi rginio 


64.0 


858 


741 


+ 117* 


183 


183 


+ 1 


Arkonsos 


38.5 


392 


330 


+ 62 


1 21 


138 


- 17 


^ A A 1/ P/> 1 lf\ f 1 I ^ 

LOOK LOunTy 


62.1 


763 


860 


- 98* 


451 


453 


- 2 


Less Thon 30 
















Son Diego 


34.4 


747 


770 


- 22 


484 


477 


+ 7 


Bol timore 


34.6 


846 


633 


+213** 


398 


442 


- 44 


Virginia 


36.0 


750 


737 


+ 13 


204 


260 


- 56** 


Arkonsos 


61.5 


484 


348 


+135** 


135 


167 


- 32* 


Cook County b 


37.9 


581 


519 


+ 62 


500 


539 


- 39 


Recipients 
















30 or Over 
















Bol timore 


42.7 


398 


394 


+ 4 


636 


632 


+ 3 


Virginia 


65.9 


432 


401 


+ 31 


411 


426 


- 15 


rU KUI1MJ d 


0 0 • i 


139 


68 


+ 71 


324 


356 


- 32 


Cook County b 


43.1 


326 


278 


+ 48 


765 


792 


- 27* 


Less Thon 30 
















Boi timore 


57.3 


527 


467 


+ 61 


621 


614 


+ 7 


Virginio 


34.1 ' 


557 


419 


+138** 


412 


453 


- 41 


Arkonsos 


6]. 9 


161 


173 


- 13 


261 


338 


* 77*** 


Cook County b 


56.9 


368 


325 


+ 43 


705 


707 


- 2 



TABLE 4.3 (continued) 



Subgroup, Wei fore 
Stotus, ond Program 


Percent 

of 
Sample 


Averoge Eornings Per Quarter 
Qucrters 4 - Lost ($) 


Averoge AFDC Poyments Per Quarter 
Quarters 4 - Lost ($) 


Fynp r Impntn 1 


Cnnt rn 1 




txpe r imenTO i 


Control 


Difference 


Recent Ul Benef its 
















Appl iconts 
















Sane 
















Son Diego 


14.1 


1270 


1304 




oyu 


A 7 1 


— ou 


None 
















San Diego 


85.9 


828 


684 


+144*** 


443 


467 


- 24 


Lobor Market 
















Appl iconts 
















Urbon 
















Virginia 


78.7 


84? 


7A9 

I O 7 


+ fin 


IOC 


*3 1 ^ 
I \ J 




Arkonsos 


64.8 


403 


302 


+ 100 


99 

7 7 


1 CO 


- 97 


Rurol 
















Virginia 


21.3 


707 


625 


+ 82 


174 


198 


- 24 


Arkonsos 


35.2 


533 


413 


+120 


167 


211 


- 24 


Qo r i nicntc 
















Urbon 
















Virginia 


78.8 


521 


421 


+100** 


411 


450 


- 39** 


Arkonsos 


56.5 


211 


120 


+ 91* 


250 


325 


- 75*** 


Rurol 
















Virginio 


21.2 


299 


346 


- 47 


413 


383 


+ 30 


Arkonsos 


43.5 


76 


152 


- 76 


326 


366 


- 41 



SOURCE AND NOTES: See Toble 4.1. 
o 

Percent of oppl iconts ond percent of recipients. 

b 

The definitions of applicant' ond "recipient* for Cook County differ from other 
programs. See text for discussion. 

C For Boltircore, Virginio, ond Arkonsos the category 'block' includes a smell nunber 
of individual s In other non-white groups. In Son Diego ond Cook County, "white* includes a small number of 
non-block, non-hisponic, non-white persons. 
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TABLE 4.5 (continued' 



T T 



Subgroup, We! fore 
Stotus, ond Program 


Percent 

of 
Sampi e 


Averoge Eorr : ngs Per Quorter 
Quorters 4 - Lost ($) 


Averoge AFDC Poyments Per Quorter 
Quorters 4 - Lost (*) 


Experimental 


Control 


Difference 


Exper imento 


1 Control 


Difference 


Ethnicity 
















Appi iconts 
















White 
















Son Diego c 


61.5 


949 


821 


+ 128* 


357 


369 


- 12 


80! timore 


33.8 


922 


767 


+155 


309 


318 


- 9 


Virginio 


41.8 


801 


663 


+138 


126 


149 


- 21 


Arkonsos 
Cook County b ' c 


16.7 


403 


259 


+144 


72 


82 


- 1 1 


21.8 


679 


709 


- 30 


353 


413 


- 61* 


Clock 
















Son Diego 


20. / 


895 


589 


+306*** 


532 


678 


-146*** 


Bol timore c 


66.2 


1035 


855 


+180** 


395 


412 


- 17 


Virginio c 


58, i 


832 


792 


+ 40 


235 


253 


- 18 


Arkonsos c 


83.3 


458 


358 


+100* 


141 


170 


- 29* 


Cook County b 


65.5 


741 


761 


- 20 


502 


517 


- 15 


Hispanic 
















San Diego 


17.8 


693 


843 


-150 


593 


556 


+ 38 


Cook County^ 


12.6 


458 


608 


-150 


510 


450 


+ 60 


Recipients 
















White 
















Bol timore 


25.1 


420 


438 


- 18 


579 


575 


+ 4 


Virginio 


26.8 


490 


398 


+ 92 


303 


305 


- 2 


Arkonsos 


8.5 


337 


55 


+282** 


207 


228 


- 21 


Cook County b 


14.3 


517 


322 


+195*** 


572 


604 


- 31 


Block 
















Bol timore c 


74.9 


490 


436 


+ 55 


644 


638 


+ 6 


Vi rgini o c 


73.2 


468 


408 


+ 60 


451 


483 


- 32* 


Arkonsos c 


91.5 


132 


136 


- 4 


290 


354 


- 64*** 


Cook County* 5 


75.3 


322 


307 


+ 15 


761 


'n 


- 11 


Hispanic 
















Cook County 


1 0.4 


324 


265 


+ 59 


730 


734 


- 4 


Applicants 
















Male 

K 
















Cook County 


10.2 


867 


931 


- 64 


432 


369 


+ 63 


Feroo 1 e 
















Cook County b 


89*8 


673 


708 


- 35 


474 


499 


- 25 


Recipients 
















Mole 
















Cook County b 


13*3 


451 


288 


+163*** 


704 


746 


- 42 


Femole 
















Cook County b 


86,7 


334 


307 


+ 27 


735 


743 


- 8 
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Information not shown in Table 4,5 is also used in the discussion where 
relevant in explaining the impact results. 
A. San Diego 

The welfare employment program in San Diego, as Chapter 2 indicated, 
was distinctive in two important respects. First, the program served only 
welfare applicants. Second, all enrollees had the same short-term sequence 
of program activities — job search followed by work experience for those 



who did not find a job. Participation rates were high for all subgroups. 

The San Diego program clearly had greater impacts on the less 
job-ready and more welfare-dependent applicants. Those with zero prior 
earnings had by far the largest earnings impacts; the welfare savings were 
spread evenly over those with zero prior earnings and those with earnings 
that were positive but under $3,000, Similarly, applicants witn a welfare 
history had most of the earnings gains and welfare savings, although both 
impacts were somewhat greater for the group with a welfare history of two 
years or less than for those with more than two years. 

Some characteristics associated with dependency other than prior 
earnings and welfare history appear to be positively related to the 
programs impacts in San Diego. The results for subgroups presented in 
Table 4.5 suggest, for example, that ethnicity and the number of children 
in a household were factors associated with greater dependency for this 
sample, as reflected in the higher welfare payments made to control group 
members who were non-white and had more than one child. 

Some of the other subgroup comparisons, however, were not consistent 
with a greater effect on the more dependent — notably the greater impacts 
for applicants who had a high school diploma or GED, a factor not usually 
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related to long-terra dependency. This result raay stem from the reliance on 
job search in San Diego. More education raay have increased the probability 
of success in a program that (unlike Baltimore) did not offer remedial 
education. 

B. Baltimore 

The Options program in Baltimore was very different from the San Diego 
initiative. Newly-mandatory AFDC recipients (with their youngest child 
just entering school) were enrolled as well as mandatory applicants. In 
addition, there was a wider range of services — from independent job 
search to education and training — and the services could vary according 
to the registrants 1 needs and preferences. Finally, enrollment during the 
period of the study was limited to 1,000 slots per year to ensure that the 
resources the planners deemed adequate to provide this range of services 
were available. As it turned out, the cost per experimental was estimated 
to be substantially higher in Baltimore than elsewhere, approximately 
$1,050 (in 1987 dollars). Participant choice was constrained by slot 
availability and guided by staff. Because the least job-ready generally 
participated at higher levels in the more intensive services — education 
and training — than did other subgroups, the subgroup impacts may have 
been influenced by the different services participants received, as well as 
by participants 1 own characteristics. 

The Baltimore results on which this study is based make use of an 

extra year of follow-up data that was not available for the MDRC final 

3 

report on the Options program. Analysis of those additional data showed 

somewhat larger earnings impacts than even the initial favorable short-term 

4 

estimates presented in the program. Earnings impacts increased after the 



first follow-up year. in fact, most of the program effects on earnings 
accrued after the first year. Welfare savings, which were small in the 
initial follow-up period, did not increase. 

For Baltimore applicants, results for the major subgroups fit the 
pattern already discussed for San Diego, The least dependent groups 
improved the least. Of additional interest, there was a relatively wide 
spread between earnings gains for applicants and for recipients. As a 
group (see Table 4.1 earlier in the chapter), applicants in the program 
earned $172 more per quarter than controls, a statistically significant 
increase of 21 percent that is comparable to the change for applicants in 
San Diego. However, recipients -- the more welfare-dependent — earned 
only $37 more. The difference in earnings impacts between applicants and 
recipients is statistically significant. 

These findings are especially important because Baltimore recipients 
had high participation rates and, as noted in Chapter 2, gross costs of 
serving a recipient were 25 percent higher than for serving an applicant. 
Recipients received a somewhat larger share of the more expensive services, 
and the follow-up period was long enough to capture the post-program 
effects of education and training. Most of the lack of impact for recipi- 
ents was due to the lack of improvement for the most- and least-dependent 
subgroups among the recipient category. Recipients with prior employment 
showed negative earnings impacts, and recipients with more than two years 
on the roils combined with no recent work experience also showed below- 
average improvement. Together, these s .oups make up most of the 
recipient sample. 

With respect to impacts for subgroups defined oy education and demo- 
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graphic characteristics, the Baltimore results were opposite to those in 
San Diego. Applicants with less education had larger than average impacts, 
perhaps reflecting the remedial education services offered by the Baltimore 
Options program. Younger women, women with younger children and women with 
fewer children also experienced somewhat larger-than-average gains among 
applicants. Such contrasting results across programs make it difficult to 
predict the effect of these characteristics under a variety of program 
settings. 

C. Virginia 

Virginia extended program participation requirements to the whole 
WIN-mandatory caseload of recipients as well as mandatory AFDC applicants. 
It also served rural as well as urban areas, and counties had considerable 
independence in implementing the program.^ Resource constraints were 
important, however. The counties relied on job search assistance as their 
principal component and on independent job search as the most widely-used 
kind of job search. Community providers, such as schools and JTPA training 
programs, which received no program funding, were utilized on a referral 
basis for education and training activities. Since controls obtained these 
education and training services on their own about as much as 
experiment als, however, it is not clear that this component contributed 
much to the estimated Virginia impacts. 6 

Impacts on employment and earnings in Virginia are interesting for the 
several anomalies they present. First, although short-term earnings 
impacts for applicants were larger than for recipients, this differential 
largely disappeared in the long-term follow-up period analyzed in this 
study. Second, applicants without recent earnings, who elsewhere had 
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average impacts or above-average impacts, experienced virtually a zero 
longer-ten?, effect in Virginia. Finally, the most job-ready subgroup, 
applicants with $3,000 or more in prior-year earnings, obtained the largest 
earnings impacts* 

To examine this constellation of results further, Figure 4.1 gives the 
quarter-by-quarter employment impacts for the three prior-earnings appli- 
cant subgroups. As shown, the experimental-control differential for the 
no-prior-earnings subgroup was relatively wide during the first six follow- 
up quarters, but then began to decline as controls "caught up." Thus, the 
short-term nature of the employment impacts, not their total absence, was 
responsible for the poor showing of this subgroup. It was also the primary 
reason why overall effects for applicants in Virginia are not greater than 
the effects for recipients there. As shown, the middle subgroup also 
experienced impact decay, although the top subgroup did not. 

It is not obvious why the Virginia program should have had its largest 
employment and earnings impacts on a top employability subgroup. But some 
consequences of that result are worth noting. Table 4.6 provides addition- 
al impact information for the subgroup with at least $3,000 in prior-year 
earnings, by showing a breakdown of the subgroup's employment status during 
the longer-term follow-up by welfare status at the end of the follow-up 
period. This breakdown covers all four possible combinations of work and 
welfare receipt: did not work and remained on welfare, did not work and 
left welfare, worked and left welfare, and worked but remained on welfare. 
The table shows an 8.4 percentage point increase in the number of individ- 
uals in this subgroup who worked after the third follow-up quarter and were 
off welfare at the last quarter (the eleventh). Corresponding to this 
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FIGURE 4.1 

VIRGINIA APPLICANTS: EMPLOYMENT IMPACTS 

. , BY PRIOR EARNINGS SUBGROUP 

Impact on 

Employment Rate (%) 
15 r 



10 - 
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TABLE 4.6 
VIRG IN I A 



AFDC APPLICANTS HAVING EARNINGS OF $3, 000 OR MORE IN THE YEAR 
PRIOk TO RANDOM ASSIGNMENT: EMPLOYMENT STATUS IN QUARTERS 
4 - LAST BY WELFARE STATUS IN QUARTER 11 



Employment ond Welfore Stotus (%) 


Exper Imentol s 


Control s 


Dlf f erence 


n o o n o earnings, k e c e i v e o no 
AFDC Poymenls ~ 


6.1 


13.5 


- 7.5* 


Hod Some Eornings, Received Nc 
AFDC Poyments 


75,7 


67.2 


+ 8.4 


Hod No Eornings, Received Some 
AFDC Poyments 


4.9 


5.6 


- 0.7 


Hod Some Earnings, Received Some 
AFDC Poyments 


13.4 


13.7 


- 0.3 


Total 


100. 0 


100. 0 


0.0 


Som pi e Size 


244 


127 





SOURCE: MDRC calculations frorr the Commonweolth of Virginia Unemployment 
Insuronce earnings records, welfjre records from the V i rgi ni a Automated Client 
Informotion System, ond Foirfcx r oi'nty AFDC r se files. 

NOTES: These doto ore regression-adjusted using ordinory leost squares, 
controlling for pre-enroll ment characteristics of somple members. There may be some 
discrepancies In calculating sums ond differences due to rounding. 

A two-toiled t-test was applied to differences between Experimental 
ond Control groups. Stotisticol significance levels ore indicated os: * = 10 
percent; ** = 5 percent; *♦* = I percent. The distributed differences are not, 
however, strictly I nde pendent. 
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increase, however, was a 7.5 percentage point decrease in the number of 
individuals who went off welfare without working. 

Thus, the effect of the program for this subgroup was to increase 
employment for those who would have been off welfare relatively quickly 
anyway. Only a minority — under 20 percent — of this subgroup would have 
remained on welfare even without special intervention, and the program had 
little impact on them. Most of the program impact evidently "spilled over - 
onto those who would only have been on AFDC for a short period anyway. As 
a consequence, welfare dependency was not affected* 

The point of this discussion is that even where programs have above- 
average earnings impacts on the most job-ready enrol lees, the result may 
not fulfill all program objectives. On the surface, the findings for 
Virginia appear to provide support for targeting the most job ready. In 
fact, they demonstrate another argument against making these individuals 
the exclusive focus of resources and attention. That is, earnings gains to 
them appear to have little effect on welfare dependency. 

Results for the subgroups defirsd by individual characteristics in 
Virginia (see Table 4.5 above) shew that, although the differences in 
impacts among subgroups were typically similar in size to those in the 
other states, they were often not in the same direction. Thus, as already 
indicated, few cross-state generalizations can be made about impact 
differences according to characteristics other than prior employment and 
welfare history. It is worth noting that the effects for applicants 
without a diploma or GED were larger than for those with one. This may be 
connected with the program's use of referral to education and training 
providers. Owing to the high incidence of similar activities among 
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controls in Virginia, however, this can be only speculation. 
D. Arkansas 

The Arkansas program was also heavily constrained by resources. The 
program formally offered job search and work experience but, in fact, rates 
of participation in work experience were low. Program cost per experiment- 
al was among the lowest of all programs in MDRC's Work/Welfare Demonstra- 
tion. 

Arkansas provides an opportunity to examine the effects of a welfare 
employment program in a low-grant state. At the start of the research, the 
maximum welfare benefit for a mother with two children was $140 per month, 
a benefit low enough to make even a small amount of earnings disqualify a 
family for AFDC. A modest increase in employment might therefore be 
expected to have a relatively large effect on welfare receipt in this 
state. As it turned out, the welfare reductions were the largest of the 
programs evaluated, whether the reductions are measured as percent on 
welfare, absolute dollar payments, or payments as a percentage of control 
group payments. 

The low benefit levels also determine the nature of the sample. 
Individuals relying on welfare were likely to have only very limited oppor- 
tunities for income elsewhere. The sample in Arkansas may therefore be 
more disadvantaged than in other states in the study. Sane evidence of 
this is the relatively low follow-up employment rates for the Arkansas 
sample compared to the other programs. 

The Arkansas sample is of special interest for one other reason. The 
state obtained federal waivers to extend mandatory program coverage to 
AFDC's whose youngest child was three to five rs old. As it turned out, 



about half the sample fell into this category. The data therefore afford 
an opportunity to see whether the presence of a pre-school child affects 
how much a woman is likely to benefit from the program. The opportunity is 
limited, however, because the research sample for Arkansas has only 1,127 
members, constraining the potential for sample subdivision. 

The earnings impacts for major employability and dependence subgroups 
in Arkansas fit the general pattern. Earnings impacts were small for 
applicants who never had their own AFDC case before and for applicants with 
year-prior earnings of $3,000 or more. Applicant returnees had the largest 
earnings impacts. The recipient subgroups had relatively weak earnings 
impacts. 

Applicants without a child under 12 and applicants who were white had. 
earnings increases somewhat above average, as in Virginia. Both these 
characteristics are associated with lower welfare dependency. Applicants 
with a high school diploma also had relatively large earnings gains. When 
these three characteristics were combined in a regression framework with 
information on other applicant characteristics, the maximum earnings impact 
occurred with the least dependent, with impacts decreasing gradually as 
dependency increased. This pattern was also similar to Virginia's in that 
the earnings gains for the least dependent were not accompanied by welfare 
reductions for them. 

In contrast to the pattern for earnings gains, longer-term welfare 
reductions were largest among recipients (Table 4.1). They were large 
among the most dependent recipient subgroups and subgroup combinations 
(Table 4.4), and exceeded the earnings increases that came to these 
subgroups during the same period of follow-up. Additional analysis 
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confirmed that the welfare savings for recipients were not the result of 

any sustained increase in employment* Employment gains for recipients 

peaked in quarter 3, and decayed after that* Most of this decay was 

associated with the absence of a program impact on the length of 
7 

employment. This raises questions about the overall financial effects on 

enrollees. For recipients, **hree quarters or more of the impact on AFDC 

payments cam.- through program effects other than a sustained increase in UI 
g 

earnings* The welfare reduction was therefore not offset by an increase 
in the er-ollees' cvii earnings. Thus, although sample members near the 
least dependent end of the continuum showed a net increase in the total of 
their own earnings plus AFDC payments, those who were the most dependent 
showed a net loss from these two sources taken together. 

This mismatch between earnings and welfare impacts among the more 
dependent was not found generally in the study samples elsewhere, but it 
nevertheless raises an important monitoring and targeting issue* Perfor- 
mance 5 ' usually thought of as tools to maximize program effects 

on ear.u J or on welfare receipt; the pattern in Arkansas suggests that 
standards should also be tested to determine whether they maximize effects 
on both outcomes at once* This issue is discussed further in connection 
with performance standards in the nex.: chapter* 

Some caument on impact differences by demographic characteristics is 
warranted* First, as shewn in Table 4*5 above, earnings gains for the more 
rural of the two study locations were lower than for Little Rock, essential- 
ly because the effects for rural recipients were negative. This is consis- 
tent with findings for the rural counties of Virginia, where recipients 
also had nega**" earnings effects (see Table 4.5 above). Evidence from an 
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MDRC evaluation of a work experience program in West Virg ^a is also 

9 

relevant in this connection. This state is largely rural and during the 
research experienced some of the nation's highest unemployment rates. The 
impact study covered primar ,n y AFDC recipients, and found virtually no 
effects on employment or earnings. The weight of this evidence is not by 
any means conclusive, but it suggests that these kinds of interventions may 
not have strong impacts among longer-term AFDC recipients in rural areas. 

Second, the welfare savings for women who were not married, particular- 
ly those who were never married, are considerably larger than for married 
or separated women. Unmarried and never-married women normally remain on 
welfare longer than married women, but the evidence from this study 
suggests that there exists a potential for reducing this greater 
dependency. As noted earlier, this finding is one of the few with some 
consistency across programs for the subgroups defined by demographic 
characteristics other than prior earnings or prior welfare. Unmarried 
women obtained virtually all the realized welfare savings in San Diego, 

Virginia, and Arkansas; of these, never-married women obtained over 70 
10 

percent. There were no welfare savings in Baltimore. The Cook County 
sample is excluded f rorr k these calculations, since marital status was not 
available for the impact sample there. 

Impact estimates for mandatory mothers with pre-school children are 
shown in Table 4.7. Average employment rates for controls indicate that 
the presence of a young child did not reduce employment in the absence of 
the program. Nor was it associated with lower impacts on employment or 
earnings. In fact, for the full sample, impacts on employment and earnings 
were larger for women with pre-school children, although this extra effect 
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TABLE 4.7 



ARKANSAS 

AFDC APFUCANTS AND RECIPIENTS: UNCONDITIONAL IMPACTS ON EMPLOYMENT, 
EARNINGS, AFDC INCIDENCE AND PAYMENTS, BY WELFARE 
STATUS AND CHILD LESS THAN SIX SUBGROUP 



Subgroup ond Wei fore Stotus 


f 

Percent 
of 

Sompl e° 


1 

Percent Employed Quorterly 
Quarters 4 - Lost (%) 


Averoge Eornings Per Quorter 
Quorters 4 - Lost ($) 


Experimental 


Control 


Difference 


Experimental 


Control 


Difference 


Any cruld Less Tho > 6 
















Ful 1 Sompl e 
















No 


46.0 


21.9 


18.3 


+3.6 


309 


254 


+55 


Yes 


54. D 


24.9 


IS.i 


+6.8*** 


342 


259 


+83* 


Appl i confs 
















No 


51.2 


27.1 


22.6 


+4.5 


404 


339 


+65 


Yes 


48.8 


34-6 


24.1 


+10.4*** 


496 


345 


+152** 


Recipients 
















No 


38.3 


11.5 


9.6 


+1.8 


125 


64 


+41 


Yes 


61.7 


14.1 


11. 1 


+3.0 


166 


161 


+ 5 



Subgroup ond Wei fore Stotus 


Percent 
of 

Sample 0 


Percent Receiving AFDC Monthly 
Quorters 4 - Lost (%) 


Averoge AFDC Payments Per Quorter 
Quarters 4 - Last (*} 


Experimental 


Control 


Difference 


Experimental 


Control 


Difference 


Any Child Less Thon 6 
















Full Sompl 6 
















No 


46. D 


32.6 


40.3 


-7.7** 


168 


207 


-39** 


Yes 


54.0 


39.5 


46.0 


-6.4** 


212 


253 


-41** 


Appl i cants 
















No 


51.2 


23.3 


28.4 


-5.0 


119 


145 


-26 


Yes 


48.8 


26.5 


30.9 


-4.4 


141 


167 


-26 


Recipients 
















No 


38.3 


50.5 


63.2 


-12.6** 


263 


327 


-64* 


Yes 


61.7 


54.5 


63.8 


- 9.4** 


295 


354 


-58** 



SOURCE AND NOTES: See Toble 4.1. 



Percent of full sompl e, oppliconts, or recipients. 
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was not statistically significant. About 64 percent of the total program 
effect on earnings came fran working with this subgroup. The differential 
appears to vary across applicant and recipient categories, but this varia- 
bility may stem fran the small number of sample points in these subdivis- 
ions of the Arkansas sample. Welfare savings were almost identical for 
women with and without a pre-school child. Once again, it is worth 
emphasizing that caution should be exercised in generalizing these findings 
owing to the small sample size and special program environment. 
E. Cock County 

In contrast to programs like San Diego and Baltimore, Cook County 
sought to involve the full WIN-mandatory population in 3ome activity. In 
the absence of substantial supplementary funding, this decision necessarily 
meant that Cook County's limited resources were spread over one of the 
largest caseloads in the nation — more than 50,000 WIN-mandatory AFDC 
recipients at any one time. The resulting program expenditure of $150 (in 
1987 dollars) per experimental was lower than any other state evaluated — 
slightly under the cost for Arkansas and less than one-sixth the cost for 
Baltimore. 

The effort to reach the full WIN-mandatory caseload with limited 
resources, combined with a belief that the best way to foster independence 
we..: to make program participants take responsibility for their own activi- 
ties, shaped the chn.dcter of the program. First, the program stipulated 
that individual clients assume the primary responsibilit\ fot finding jobs 
on their own. Independent Job Search was selected as the primary activity, 
although individuals who completed US without finding a job could then be 
required to participate in work experience for three months. Second, 
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program staff, whose caseloads averaged about 300, were evaluated largely 
on the basis of the AFDC grant reductions they achieved. They therefore 
tended to concentrate more on administrative and monitoring functions than 
on direct services to clients? sanctioning for failure to satisfy program 
requirements, for example, was more automatic than in other programs 
studied by MDRC. In sum, the balance between enforcing obligations and 
providing services weighed more heavily towards the former in Cook County 
than elsewhere. 

As noted, data distinguishing between applicants and recipients were 

lacking in Cook County. As a substitute, individuals who received no AFDC 

payments during the three months preceding randan assignment are considered 

applicants; the remainder (about two-third of the sample) are considered 
... 12 

recipients. As also noted, this limitation alono would probably not 

reduce comparability with the other program subgroups substantially. The 
important difference is that in Cook County, only approved applicants and 
recipients were enrolled. For this reason, applicants in Cook County had 
higher rates of continuing welfare receipt than applicants in other program 
samples, falling about midway between the other applicant and recipient 
samples. 

Other adjustments in subgroup definitions were also necessary for Cook 
County. As stated earlier, neither background demographic nor welfare 
history data were available. AFDC grant payments were available for a 
ten-month period preceding randan assignment. These were used to subdivide 
the large category of recipients with no prior-year earnings. The first 
subdivision was between those who did not receive welfare payments in all 
ten prior months and those who did. This latter groups was further divided 




into three groups depending on payment amount • 

The sample was thereby broken down into eight mutually exclusive 

categories, shown in Table 4.8 in descending order of welfare dependency. 

The table gives earnings and welfare impacts for these subgroups. As 

stated in the final evaluation report for the Cook County program, there 

were no statistically significant employment and earnings impacts for the 

13 

full sample, although there were small welfare savings. The table indic- 
ates that increases in employment and earnings were obtained for one 

subgroup in the middle range of dependency, namely, recipients with prior 
14 

employment. This subgroup accounted for 13 percent of the sample, but 
for virtually all the earnings impact and 35 to 40 percent of the aggregate 
welfare savings. Quarterly earnings gains for them were nearly $150 larger 
than their AFDC payments reductions. Neither the subgroups above them on 
the dependency scale, nor the large body of individuals in the lower half 
of the dependency scale gained much from the program. 

Of the demographic characteristics, only age, sex and highest grade 
obtained at time of welfare application were available (Table 4.5). 
Consistent with the earnings gains observed for recipients with prior 
earnings, Table 4.5 shows statistically significant earnings impacts for 
recipients with a high school diploma, and also shows relatively large 
earnings gains for the small subgroups of recipient males and recipient 
whites. These were all less-dependent subgroups within the recipient 
category. 

III. Summary for the Major Subgroups and Combinations 

By way of summary, Table 4.9 displays the earnings and AFDC payment 
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TABLE 4.8 



COOK COUNTY 



AFDC APR. I CANTS AND RECIPIENTS: UNCONDITIONAL IMPACTS ON EMPLOYMENT, EARNINGS. AFDC INCIDENCE 
AND PAYMENTS. BY WEL FARE STATUS. PRIOR EARNINGS. AND PR IOR WELFARE PAYMENTS 



Subgroup, welfore 
diQiUd* one rrogrcm 


t 

Percent 

of 
SampI e 


Percent Employed Quorterly 
Quorters 4 - Lost (%) 


Averoge Eornings Per Quorter 
Quorters 4 - Lost ($) 


Experiment! Control 


Difference 


Experiments 


Control 


Difference 


Appl iconts b 
















Pr 1 nr Cn rnlnnc 

• r i or torni ngs 
















$3000 or More 


11.1 


^ Q 
90 . 7 




+ 1.7 


1373 


1336 


+ 37 


$1-2999 


6.9 


OO . 9 


JO. 4 


- 4.8 


555 


722 


-167 


None 


15.7 


15.0 


16.5 


- 1 .5 


268 


301 


- 33 


Recipients** 
















Some Prior Eornings 


13.1 


42.2 


38.0 


+ 4.2** 


865 


685 


+179*** 


No Prior Eornings 
















1-9 Months Prior AFDC 


6.8 


15.7 


14.3 


+ 1.4 


252 


222 


+ 30 


10 Months Prior AFDC 
















Averoge Poyment 
















Less Thon $3000 


18.8 


15.8 


16.5 


- 0.7 


242 


284 


- 42 


$3000-3499 


12.5 


13.4 


11.6 


+ 1.8 


219 


196 


+ 24 


$3500 or More 


15.1 


12.8 


9.3 


+ 3.5* 


188 


125 


+ 63 



Subgroup, Welfore 
Stotus, ond Program 


Percent 
of 

o 

Sompl e 


Percent Receiving AFDC Monthly 
Quorters 4 - Lost {%) 


Averoge AFDC Payments Per Quorter 
Quorters 4 - Lost ($) 


Experimental 


Control 


Difference 


Experimental 


Control 


Difference 


Appl iconts b 
Prior Eornings 
$3000 or More 
$1-2999 
None 


11.1 
6.9 
15.7 


46.4 
56.7 
59.4 


47.0 
59.3 
62.8 


- 0.6 

- 2.6 

- 3.4 


406 
460 
519 


403 
493 
540 


+ 3 

- 33 

- 21 


Recipients b 
Some Prior Eornings 


13.1 


62.1 


67.0 


- 4.8** 


567 


607 


- 40* 


No Prior Eornings 
1-9 Months Prior AFDC 


6.8 


66.8 


68.7 


- 1.9 


613 


620 


- 7 


10 Months Prior AFDC 
Averoge Poyment 
Less thon $3000 
$3000-3499 
$3500 or -More 


18.8 
12.5 
15.1 


76.1 
83.1 
85.5 


77.0 
82.6 
86.1 


- 0.9 
+ 0.5 

- 0.7 


601 

si; 

1020 


596 
820 
1041 


+ 5 

- 6 

- 21 



SOURCE AND NOTES: See ^ble 4.1. 



Percent of ful I sample. 




The definitions of 'applicant' and 'recipient' for Cook County differ from the other 
'oms. See text for discussion. 



TABLE 4.9 



AFDC APPLICANTS AND RECIPIENTS: SUMMARY OF IMPACTS 
FOR MAJOR SUBGROUPS AND COMBINATIONS 





Quorter 


y Eornings 


Impoct, Quarters 4 - Lost ($) 


Subgroup 0 


Son D i ego 


Bo 1 1 imore 


Virginia 


Arkonsos 


Cook 
County 


First Tier 












App 1 iconts with No 
Prior AFOC 


+ 37 


+ 121 


-13 


+ 26 




Appl iconts Wi th $3000 
or More Prior Eornings 


-39 


+ 18 


+307*** 


+ 12 


+ 37 


Second Tier 












App 1 icon t Re turnees 


+158** 


+188*** 


+ 114* 


+211 *** 




Appl iconts With Less Thon 
$3000 Pr i or Earn ings 


+181 *** 


+244*** 


-16 


+121** 


-75 


Appl icont Returnees With 
Less Thon $3000 Pr i or 
Eo rn ings 


+151** 


+253*** 


+ 20 


+202** 




Third Tier 












Al I Rec i p ien t s 





+ 37 


+ 69* 


+ 19 


+ 46** 


Rec i p ien ts W i th More Thon 
Two Yeors on AFDC 


_ — 


-0 


+1 10** 


+ 14 




Rec i pien ts with No 
Prior Eornings 


... 


+ 1 04** 


+ 70 


+ 29 


+ 12 


Rec i p ient s With No Prior 
Eornings ond More Thon 
Two Yeors on AFDC 




+ 88 


+ 94 * 


+ 28 




All AFDC 












Quorterly Eornings Impoct 


+118** 


+96*** 


+ 72** 


+ 70** 


+ 19 


Ave roge Control -Group 
Earn ings 


773 


634 


541 


257 


451 



(cont inued ) 
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TABLE 4.9 (continued) 





Quorterl y 


AFDC Poyment 


1 mpoc t , 


Ouorters 4 - 


Lost U) 


Subgroup 0 


Snn n i pnn 
u ii u i c y u 


Dn 1 + J flirt f D 


V i r g i n i o 


A rko nso s 


Cook . 
County 


First Tier 












App 1 iconts With No 

Prinr A P n r 

r I 1 Ul ArUL 


-5 


-9 


-28 


-31 




Ad d i iconts with J^nnn 
or More Prior Eornings 


+ 3 


+ 7 


+ 8 


+ 1 


+ 3 


Second Tier 












App 1 icon t Re turnees 


-47 


-15 


-16 


-19 





Appl iconts With Less Thon 
$3D00 Prior Eorn ings 


-48* 


-23 


-31 


-30* 


-25 


App 1 icon t Re turnees Wi th 
Less Thon $3000 Prior 
Eornings 


-63* 


-19 


-29 


-22 





Third Tier 












Ai i Rec i p ien t s 





+ 5 


-24 


-6D*** 


-13 


Rec i pien ts w i th A\ore Thon 
Two Yeors Dn AFDC 





+ 19 


-48** 


-44* 





Rec i p ien t s With No 
Pr i or Eo rnings 





+ 1 


-26 


-63*** 


-6 


Rec i pien ts w i th No Pr i or 
Eorninos ond More Thnn 
Two Yeors on AFDC 





-1 


-48** 


-4 8* 





AI 1 AFDC 












Quorterly AFDC Poyment Impoct 


-33 


-5 


-23* 


-4 0*** 


-n 


Averoge Cont rol -Group AFDC 
Poyment s 


469 


501 


315 


232 


646 



SDURCE AND NDTES: See Toble 4.1. 



Subgroups within eoch tier overlop, constituting olternotlve woys 
of grouping individuols. There is olso overlop between first ond second tiers. 

b 

The definitions of 'opplicont* ond 'recipient* for Cook County 
differ from the other progroms. See text for discussion. 
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impact estimates for several of the major subgroups and combinations 
discussed. As can be seen, these subgroups are not mutually exclusive. 
But they are useful because they represent several ways that program 
planners could distinguish among program eligibles using information about 
prior work and welfare. The table is organized in three roughly increasing 
tiers of dependency. Several conclusions may be drawn. 

First, subgroups in the top tier (the least dependent subgroups) do 
not generally exhibit above-average employment and earnings impacts; 
indeed, they usually exhibit low impacts, if any. These subgroups 
constitute anywhere from 10 to over 50 percent of applicants. Any program 
design that focuses the bulk of resources on these subgroups, passing over 
subgroups further down, is unlikely to be maximizing program impact. 

The second tier (the middle dependency tier) consists of applicants 
either with sane welfare history or with a weak work record. Earnings 
impacts for these subgroups were more likely to be above average, although 
this was not tri^ for all subgroup samples in this tier. Across samples, 
applicant returnees evidenced the most consistent gains of any subgroup in 
this study. In the four programs where welfare history was available they 
accounted for 65 to 70 percent of the average earnings impact for all 
groups, although it is unlikely that this subgroup obtained impacts in the 
fifth program (Cook County).* 15 These returnees were individuals who had 
shown some dependence on AFDC in the past, but who had not received it 
continuously. This may derive from a greater capacity for independence, 
with returns to welfare occasioned by situational difficulties or loss of a 
job — problems that can often be alleviated by the kinds of work/welfare 
programs included in the study. 
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The third tier (the most dependent subgroups) comprises several 
recipient subgroups. These groups together make up a large share of program 
eligibles — 40, 50, and 60 percent of the samples, respectively, in 
Arkansas, Baltimore, and Virginia, and two-lnirds of the sample in Cook 
County. Employment and earnings impacts were found for seme recipient 
subgroups, but the pattern was not consistent. Lack of impacts was not due 
to lack of program participation. On the contrary, participation rates were 
as high as or higher for recipients than for applicants. 

This third tier does, however, cover a rather wide range of depend- 
ency. The lower part of that range is composed of recipients without 
recent employment, with more than two years previous welfare receipt, or 
both. For the four recipient samples, only occasionally did any of these 
subgroups attain above-average earnings impacts, although seme were 
positive and statistically significant. Therefore, exclusive targeting on 
the most dependent may be a risky strategy for a program whose major 
objective is to maximize earnings impacts, at least for the kinds of 
programs under study. 

Welfare savings, because they were smaller than earnings increases, 
are more difficult to contrast across subgroups. The least dependent 
groups showed no welfare impacts. For the other two tiers, no strong 
evidence emerged that favored preferential targeting f^t any subgroup, 
although the more dependent subgroups of the samples tended to show larger 
welfare effects than the other subgroups in some cases. 

No subgroup emerges clearly and consistently as the most promising 
group on which to focus exclusive targeting efforts when sufficient 
resources are available to serve the bulk of program eligibles. On the 
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other hand, subgroups in the second tier shov promise for priority 
attention when earnings gains are the goal and scarce resources are a 
constraining factor. Applicant returnees evidenced the most consistent 
earnings responses, with gains in all four of the programs that produced 
overall earnings impacts. Applicants in the lower prior earnings brackets 
and returnees with lower prior earnings are overlapping categories which 
had higher than average earnings impacts in three of five programs. 
Program operators who can serve only a portion of the KIN-mandatory 
caseload may consider one of these subgroups a suitable starting point. 
The results indicate that exclusive attention to the top tier is clearly a 
poor strategy if an increase in earnings or a reduction in dependency are 
program objectives. 

The evidence further suggests caution in targeting very narrowly. 
Applicants as a whole obtained benefits in four of five program samples; 
progressively narrower targeting within the applicant group does not 
necessarily yield progressively larger impacts. Operators should also be 
sensitive to the possibility that the presence of seme job ready regis- 
trants may itself help program effectiveness by providing encouragement to 
the less job ready and to program staff. 

Interpretation of the findings for recipients is problematic. The 
findings indicate that restricting program enrollment only to recipients 
would be a risky strategy for programs whose objective was to maximize 
earnings impacts. However, a substantial share of the estimated welfare 
savings were found among recipients. It remains an open question whether 
shifting resources from the least dependent in order to provide more 
expensive services to the other subgroups would have increased earnings 
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impacts on those subgroups. 

The combination of impact differences across dependency subgroups and 
according to demographic characteristics suggests a possible threshold 
ef£ect for earnings. That is, for cases whose multiple disadvantages 
combine to make them more dependent than some threshold level, the 
typically low-cost services provided by the programs in this study may 
begin to lose their effectiveness in raising earnings. The data in support 
of the threshold effect hypothesis are only suggestive — other character- 
istics are associated with dependency, and some dependent subgroups did 
relatively well in some programs. 

Nevertheless, it is at least plausible that there is a substantial 
group for which the relatively lew-cost interventions included in this 
study lose sane of their effectiveness in improving earnings outcomes. 
This group may include recipients with more than two years on AFDC, with no 
recent earnings, and without a high school diploma. This combination of 
characteristics was associated with below-average earnings gains not only 
in Baltimore, but also in Arkansas and, to a lesser degree, in Virginia. 
San Diego is not comparable because the evaluation did not include 
recipients. No information about length of prior welfare was available in 
Cook County, but such a threshold effect is suggested by the finding that 
recipients without prior employment had no earnings impacts. 



CHAPTER 5 



MEASURES OF PROGRAM PERFORMANCES 

Our analysis of welfare program performance for different subgroups of 
the population defines performance in terms of program impacts on employ- 
ment and welfare of the people it serves. But direct estimates of impacts 
cannot be obtained cheaply or quickly enough to be used in the management 
of most programs. The purpose of this chapter is to assess the validity of 
several performance measures by examining the correlation between such 
measures and the program impacts discussed in Chapter 4. This chapter has 
two sections. The first section discusses job-entry and off-welfare 
measures. For all indicators, both unweighted and weighted versions are 
formulated. The unweighted ones count all enrollees equally; the weighted 
ones permit more weight to be given to program success in dealing with the 
less employable, more dependent subgroups. The second, and last, section 
addresses program participation and coverage indicators. 

I* Job-Entry and Off-Welfare Measures 

It is often argued that counting job entries and welfare case closures 
focuses the attention of program managers on the true program objectives, 
namely, increasing employment and decreasing welfare dependency. Emphasis 
on recording job entries can also serve an administrative function in AFDC 
eligibility determination, by encouraging and providing program staff with 
an incentive to make sure that client earnings are accurately reported to 
Income Maintenance Offices and that grant reductions are made where 
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appropriate. 

These outcomes are only valid indicators of underlying performance, as 
noted, if they are, in fact, correlated with real program impacts. If 
there is such a correlation, a high rate of job entry would indicate a 
large impact on employment, and a lew rate would indicate a small impact, 
and similarly for AFDC case closures. 

A. How Useful Are the Outcome Measures? 

Several empirical observations already discussed strongly imply that 
the correlation of outcomes with impacts may not, in fact, be a strong one. 
For one thing, as the experience of the control groups in this analysis has 
shown, many recipients find jobs and leave welfare in the absence of 
program assistance. Most employment and welfare departures recorded for 
the experimental groups, in other words, would have occurred without the 
special intervention. Outcome measures of performance, therefore, clearly 
overstate true performance, and the overstatement is quite large. Thus, 
cperators of welfare employment programs could record substant J al numbers 
of 'placements' and large 'welfare savings' w *hout providing any real 
services at all and without changing behavior in any way. 

In this regard, it is telling that within the experimental groups 
examined for this study at least as much employment and as many vclfare 
case closures accrued to nonpar ticipants as *-o those who actually 
participated in services. 1 This does not mean the programs had larger 
impacts on nonparticipants than participants, or even that they had any 
impacts on nonparticipants. It means only that substantial numbers were 
able to find jobs with minimal assistance and encouragement. 

Simple outcome measures also cannot provide meaningful comparisons of 
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performance across programs or components or from year to year. Some 
programs' high rates of job entry may result from their having a relatively 
■job-ready" target population or a strong labor market; the apparently poor 
rate of job entry for other programs may stem fran a less job-ready target 
population or a poorer labor market. Because San Diego enrolled only 
applicants and Cook County enrolled , urrent recipients and approved 
applicants, their rates of job entry were destined to be different from 
each other, even had they implemented identical programs. Follow-up 
employment rates for experimental s in each of the five programs studied 
differed substantially, but those differences did not reflect primarily 
differences in program impact. Mo. . of the differences were determined by 
the subgroup composition of the enrollees, by local AFDC eligibility 
regulations^ and by area labor market conditions. With the exception of 
Cook County, program impacts on emplr/ment were similar across programs. 
In other words, a lower post-program employment rate did not signify a 
lcwer program impact. 

For these two reasons — overstatement and misleading cross-program 
comparisons — simple outcome measures do not fulfill the monitoring 
function required of a valid indicator of performance. Quite as serious as 
either of these problems is the fact that the degree of overstatement of 
performance differs substantially for different subgroups of program 
eligibles. In the preceding chapter, it was shown that individuals in the 
top eroployability categories typically experience post-program employment 
rates two or even three tiroes a< large as groups in the lower categories. 
The dramatic differences in job entry rates that this implies provide a 
strong incentive for program operators to pay the most attention to 
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individuals at the top. Conscientious program administrators seeking high 
job-entry rates may focus staff time and resources on placing relatively 
job-ready registrants, many of whom might have been able to find jobs on 
their own. Administrators are given no incentive to test or implement 
services that would be effective for the least job-ready subgroups. This 
consideration can be even more important for voluntary programs than for 
mandatory ones, since outreach and screening have greater scope in 
determining the size and composition of the enrolled population when 
participation is not required. 

An additional examination of the correlation between outcome measures 
and impacts was undertaken with the experimental Work/Welfare data. For 
this purpose, a short-term j ob entry was defined as employed at sane point 
during quarters 2 or 3 after randan assignment, and short-term off-welfare 
status was defined as receiving no welfare payments in the third quarter. 
Somewhat longer-term measures took into account quarter 4 and the following 
ones for employment, and quarter 6 for welfare payments. It should be 
noted that job entry rates derived from UI earnings will be higher than the 
actual job entry rates reported by these programs because the UI data 
provide a more complete record of employment than the contact information 
available to program staff. 

Table 5.1 displays, in sunmary form, the results of correlating the 
short-term job entry and off-welfare outcomes with program impacts on 
earnings and welfare payments estimated for each experimental group member 
on the basis of regression results for San Diego, Baltimore, Virginia, 
Arkansas, and Cook County. 3 The indicators are ranked in the table as 
follows: 
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TABLE 5.1 



AFDC EXPER IMENTALS : VALIDITY OF SIMPLE JOB ENTRY 
AND OFF-WELFARE PERF0RA1ANCE INDICATORS 



D r A A m /\ mm 

r l Oy 1 Oil 


Correlotion of indicolor 
JOB ENTRY 
with impoci on: 


Eorn i ngs 60 in 


Welfore Sovings 


Son Diego 


poor 


weok 


B 0 1 t 1 mor g 

t> \y 1 1 fill VI v 


weok 


weok [foir] 


Vi rg inio 


GOOD 


poor 


Arkonsos 


GOOD 


weok 


Cook County 


GOOD 


f 0 i r 



Progrorii 


Correlotion of indicotor 
OF;- WELFARE 
with impoct on: 


Eorn i ngs Go i n 


Welfore Sovings 


Son Diego 


foir 


weok 


Boll imore 


foir 


foir 


V i r g inio 


foir 


poor 


Ark on sos 


foir 


foir 


~ook County 


GOOD 


fo ir [GOOD? 



SOURCE: 



See Tob I e 4.1 



NOTES: This toble summorizes the correlations between the desic 
noted jndicotor and earnings goins or welfore sovings. The following 



symbols ore used 



GOOD 


correl 


0 t i on 




s i gn i f 


: c on t 


foir 


corret 


c t i 0 n 




s i gn i f 


icont 


weok 


correl 


0 1 ion 




s i gn i f 


i con t 


poor 


cor r e 1 


0 1 i on 




t con t 





A longer-term version of the indicotor was also tested in 0 second 
procedure by exomining its partial correlotion with the predicted impoct 
while controlling for the short-term indicotor. If the partial correlotion 
of 0 longer-term version roised the indicator's ronk from one of the two 
lower to one of the two higher rotings, or from 'foir' to 'GOOD,' thot 
chonge is noted in brockets in the toble. 'Short-term' ond •longer-term' 
indicotors ore defined os follows: 



3hort-term job entry Any Ul earnings quorters 2 or 3 

Longer-term job entry Any Ul earnings quorters 4 through lost 

Short-term off welfore No AFDC poyments quorter 3 

Longer-term off welfore No AFDC poyments quorter 6 
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Rating 



Correlation between indicator and impact 



GOOD 



positive, statistically significant 



FAIR 



positive, not statistically significant 



WEAK 



negative, not statistically significant 



POOR 



negative, statistically significant 



Rankings are provided for all short-terra versions of these indicators. If 
the longer-term version indicated substantial improvement, the higher rank 
is sho«n in brackets. It should be noted that a result of "good' means 
only that the indicator has some validity when applied to program enrollees 
as a group ; it does not imply that the measure is a reliable indicator that 
the program has or has not had an effect for any individual enrollee. 

The ideal indicator would have a good rating with both earnings gains 
and welfare reductions. The actual correlations fell short of this ideal. 
For two of the programs (San Diego and Baltimore), the job-entry measure 
was clearly not consistent >-ith actual performance; short-term job entry 
was a weak or poor indicator of earning impacts , and the longer-term 
version showed little impiONrement. Job entry was also rot a satisfactory 
indicator of welfare savings, nor was the off-welfare measure. 

For the other three programs — Virginia, Arkansas and Cook County — 
job entry had good ratings for earnings impacts. However, only for Cook 
County was there any consistency between the correlation of job entry with 
both earnings and welfare reductions, and even there the ratings were not 
identical. For Virginia and Arkansas, job en .ry was a poor or weak 
indicator of welfare reductions. This is consistent with the subgroup 
findings: Virginia and Arkansas achieved their largest earnings impacts at 
or near the top of the dependency spectrum, whereas welfare reductions 
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occured primarily among sample m mbers further down the dependency 
spectrum, where job entry rates were lower. 

The short-term off-welfare measure was slightly better. It had fair 
ratings as an indicator of earnings gains in four of the programs and a 
rating of good in Cook County. It had a poor rating as an indicator of 
welfare savings in Virginia, but reached the level of fair for three of the 
five programs (Arkansas, Cook County, and Baltimore). 

Interestingly, longer-term data did not lead to much better results. 
In only 2 of the 20 correlations in Table 5.1 did the longer-term version 
of an indicator improve the correlation. It should be noted, however, that 
the "short-term* outcome measures tested here make use of follcw-up which 
is already longer than that available to many program operators, who often 
have only the enrollee's status at date of termination from the program. 

These empirical correlations are not the only test of indicator 
validity, but they do highlight sane of the disadvantages of simple job- 
entry and off-welfare performance measures. They confirm that outcome 
indicators are not always strongly correlated with underlying program 
impacts. They also indicate that, even in cases where job entry was 
correlated with earnings impact, this performance measure produced weaker 
results for reduction of dependency, a problem which concurrent use of case 
closure standards seems unlikely to fully correct. Job-entry standards 
were most compatible with earnings gains accruing to individuals who would 
nave u-rn off AFDC soon anyway. For administrators who wish to affect 
individuals who ordinarily would stay in the welfare system longer, simple 
outcome standard? would not appear to encourage optimal allocation of 
program resources. 
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Up to this point, the job-entry measures have given equal value to all 
WIN-mandatory clients, regardless of their work and welfare histories. The 
disadvantage of such unweighted measures is that they contain incentives 
for program operators to serve clients unequally. This built-in bias is 
not necessarily in the direction of maximum impact, nor does it always lead 
to coordinated earnings gains and welfaro savings. To reduce these 
problems, it is appropriate to consider a different scoring strategy -- one 
that gives more weight to job entries for registrants with weaker previous 
work records or longer time on welfare. The rationale for weighting is to 
retain the best property of outcome measures — their emphasis on employ- 
ment and departure from welfare as program goals — while overriding their 
undesirable allocation properties. 

Weighting may be simple or complex. It may mean simply keeping 
separate track of job entry rates for applicants and recipients or for 
individuals with high and low prior earnings. Administrators may then set 
different job entry standards for each subgroup and announce a higher 
priority (i.e., greater weight) on achieving job entries for one over the 
other. Or weighting may be based on a complex formula, involving a long 
list of eiirollee characteristics, with the weights derived from regression 
analysis of employment and welfare behavior. Yet another kind of weighting 
takes the form of "waiting' — providing a low-cost job search component to 
all enrollees to begin with and reserving more intensive services for indi- 
viduals who have still not become employed after several weeks or months. 

To find out whether weighting might improve indicator validity, the 
correlations in the preceding section were run on weighted job-entry and 
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off-welfare measures. These new performance measures were based on a 
number of different weighting schemes? but all gave greater weight to 
successful outcomes for sample members with low predicted future earnings 
or long predicted length of time on welfare. Some of the tested weighting 
schemes used complex, regression-based indices of predicted earnings and 
welfare receipt of the kind already discussed. The poor correlations in 
San Diego ai;d Baltimore improved for some of these weighting schemes, 
providing some evidence that giving extra weight in setting performance 
standards to the job entries to less employable welfare lecipients may 
improve the link between performance measure and impact. 

The most complex weighting systems utilize a complete demographic 
profile for each enrollee and assign a weight for each characteristic. 
This maximum use of available information may seem desirable, since it 
provides the most detailed weighting scheme. This level of detail may well 
be suitable for evaluations at the state and national level — where addi- 
tional weights can be calculated for local labor market conditions and AFDC 
statutory grant levels. It has drawbacks as a tool for local operators and 
caseworkers, however. First, the extra data collection is costly, and the 
more data needed the higher the chance of error. Second, the weights them- 
selves must be estimated with care so as not to over-emphasize a variable 
that actually has relatively little operational importance. Third, and 
perhaps most important, the complexity of the formula may obscure rather 
than clarify the operational priorities line staff need. Because of these 
disadvantages, it is important to see whether the sought-after improvement 
in outcome indicators can be achieved by simpler weighting alternatives. 

An alternative approach uses information about only the two best 
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predictors of future employability and dependency — namely, prior 
employment and welfare experience. One such measure can be created for job 
entries based only on prior employment: 4 



$3,000 or more earnings in prior 
$1-2,999 earnings in prior year: 
Not employed in prior year: 



year: 1 point per job entry 

2 points per job entry 
4 points per job entry 



Another measure applies the same weights to length of prior AFDC receipt: 



Never had own AFDC case: 

Had own AFDC case for two years 

Had own AFDC case more than two 



1 point per job entry 
or less: 2 points per job entry 

years: 4 points per job entry 



These or similar measures might be applied airectly in scoring performance 
of program staff or local program units. The same principle would be 
embodied more flexibly in a directive to grant service priority to the 
lower subgroups of each set, with proper administrative controls to assure 
^cmpliance. 

The correlations of the two weighted measures with impacts on earnings 
and welfare payments are summarized in Tables 5.2 and 5.3, along with the 
corresponding increases or decreases in validity relative to unweighted 
measures. Increases in validity were recorded for 14 out of the 36 short- 
term correlations, decreases for three. These weighted indicators thus 
represented a moderate improvement over unweighted measures. For earnings 
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TABLE 5.2 



AFDC EXPE R IME NTALS : VALIDITY AND IMPROVEMENT OF JOB ENTRY 
AND OFF-WELFARE PERFORMANCE INDICATORS 
WE IGHTED BY PRIOR EARNINGS 





Correlotion ond improvement 0 of indicator 
JOB ENTRY . 
WEIGHTED by PRIOR EARNINGS 
with impoct on: 


Progrom 


Eorn ings Go in 


Wei fore 


Sov i ngs 


Son Diego 


GOOD + 


GOOD 


+ 


Bo 1 1 j more 


GOOD + 


foir 


+ 


Vi rg Inio 


poor - 


weok 


0 


Arkonsos 


GOOD 0 


weok 


0 [foir +] 


Cook County 


GOOD 0 


GOOD 


+ 





Correlotion ond improvement 0 of indicotor 
OFF WELFARE h 
WEIGHTED by PRIOR EARNINGS 0 
with i mpoc t on : 


Progrom 


Eorn i ng s Go i n 


We 1 fore 


Sov i ngs 


Son Diego 


GOOD + 


GOOD 


+ 


Bol t imore 


GOOD + 


GOOD 


+ 


Vi rg inio 


poor - 


fo ir 


+ 


Arkonsos 


foir 0 


foir 


0 


Cook County 


weok - 


GOOD 


+ 



SOURCE: See Tob le 4.1. 

NOTES: See Table 5.1. 

0 Improvement over the unweighted version of the indicotor is 
Shown os foil ows : 

+ correlation increased from poor or weok to foir or 
good, or from foir to good 

correlotion decreased from foir or good to poor or 
weok, or from good to foir 

0 no chonge or mini mo I change 



^Weights were ossigned to |ob entry ond off-welfare scores op 
the bosis of prior earnings os follows: 

Prior-Yeor Eornings Points 

$3000 or More 1 
$1-2999 2 
None 4 
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TABLE 5.3 

AFDC EXPERI MENTALS : VALIDITY AND IMPROVEMENT OF JOB ENTRY 
AND OFF-WELFARE PERFORMANCE INDICATORS 
WEIGHTED BY PRIOR WELFARE 





Correlot ion ond imp 
JOB 

WEIGHTED by 
with i 


rovemenl 0 of indicotor 
ENTRY 

PRIOR WELFARE 0 
mpocl on: 


Progrom 


Eorn ings Go in 


Wel fo^e Sovings 


Son D i ego 


weok 0 


poor 0 


Bo 1 t imore 


poor 0 


poor 0 [-] 


V i rg i n i o 


GOOD 0 


weok 0 [GOOD +] 


Ark on s o s 


GOOD 0 


weok 0 


Cock Counly c 







Pr ogrom 


Correction ond imp rovemenl 0 of indicotor 
OFF WELFARE 
WEIGHTEO by PRIOR WELFARE 0 
with i mpu^ t on : 


Eorn ings Go in 


We I fore 


Sov ings 


Son 0 i ego 


GOOD + 


poor 


0 


Bo 1 f imore 


foir 0 


foir 


0 


V i rg i n i o 


GOOD + 


weok 


0 


Arkonsos 


GOCD + 


f o i r 


0 


Cook County c 









SOURCE : 7ee Tob I e 4.1. 
NOTES: See TabJe 5.1. 

0 Improvement over the unweighted version of the indicotor is 
shown as f o I lows : 

+ correlation increosed from poor or weok to foir or 
good, or f ron, foir to good 



ccrrelotion decreosed from fo 
wee*;, or from good to foir 

no change or mini mo I chonge 



or good to poor or 



. ^Weights were ossigned to job entry ond off-welfore scores on 

the bosis of prior welfore history os follows: 



Hod Own AFDC Cose 
Neve r 

Two Yeors or Less 

More Thon Two Yeors 



Points 

1 

2 

4 



Cook County. 



c Length of prior welfore information wos not ovoiloble for 
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impac s, the weighted measures predominantly had a rating of good. Much of 
the improvement came for San Diego and Baltimore, particularly with the 
prior earnings weights. For Virginia, Arkansas, and Cook County, weighted 
indicators usually ranked as high as unweighted, but not higher, although 
they would still be preferable in that they would tend to counteract the 
adverse targeting properties of the unweighted measures. 

.lighting also made a difference to the correlations with welfare 
savings. In particular, off-welfare measures weighted by prior earnings 
showed good or fair correlations with welfare savings in all five programs. 
In addition, in all but two cases weighted off-welfare measures had 
correlations with earnings gains that were good or fair. This suggests 
that AFDC receipt data, which is accessible to social services agencies, 
may usefully supplement or substitute for employment data, which is often 
not available and, if available, not complete. 

Weighting improved the indicators more than did extending the follow- 
up dat i included in the measures. For the simple weighted indicators 
shown, longer-term information only occasionally increased the correlations 
and only once changed a rank from fair to good. And only for 2 of the 36 
correlations did the addition of longer-term data lead to further 
improvement in indicator validity than the weighting. 

To assess the consistency of job entry performance measures with 
program cost-effectiveness, further analysis was undertaken with the 
subgroup cost data available for San Diego and Baltimore. Estimates of 
total gain to government budgets were produced from estimates of welfare 
and Medicaid reductions and increases in taxes paid, minus program costs. 
Total gains for program enrollees were estimated by subtracting welfare and 
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Medicaid reductions and any increases in taxes from increases in earnings. 
Job entries with simple prior earnings weights were found to be positively 
correlated with both total gains for government budgets, and total gains for 
program enrollees. 

Overall, then, weighting job entries by prior earnings or prior 
welfare receipt appears to constitute an improvement over the unweighted 
measure, it is worthwhile also to speculate on extensions of the weighting 
principle studied here. Additional objective factors (such as the absence 
of a high school diploma) might be given weight, as well as other factors 
relevant in Particular local circumstances and program goals. Weighting 
could also be applied to other outcome measures, including wage rates and 
job retention. The weighting concept could be used to economize on data 
costs by assigning higher priority to collecting employment information for 
enrollees with lower prior earnings or a longer welfare history. In this 
connection, where a subsampling strategy is employed for the collection of 
f/sjat-up data it is probably a good idea to make sure that the more 
dependent and less employable are sampled in statistically adequate 
numbers. 

How much accuracy is required in setting relative weights? The 
research conducted for this study suggests that selecting appropriate 
characteristics for weighting is probably more important than precision in 
the weights themselves. Weighting by prior earnings worked sanewhat better 
than weighting by AFDC history, but both are undoubtedly key predictions of 
future experience for WIN-ntandatory AFDC caseloads. It would be difficult 
to jurlxfy a weighting scheme that ignored both these variables. 

The patterns of rankings vary relatively little when the weight values 
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changed. For example, when the 1-2-4 prior earnir* s weights were replaced 
with weights of 1-2-8 and 1-4-4, similar, though not identical, rankings 
emerged. Broad program directives, such as giving placement priority to 
enrollees with more than two years of AFDC history, are consistent with the 
weighting concept tested here. And, questions of technical validity aside, 
differential weighting may be seen as a method of increasing the incentives 
for serving more dependent subgroups o£ the WIN-mandatory caseload on 
grounds of equitability as well as efficiency. For this purpose, 
"accuracy" of weighting has a value dimension as well as a technical one, 
and judgmental adjustments to weights may therefore be seen as legitimate. 
Weights derived from regression analysis of a nationally-representative 
sample sidestep the political process, which is seen as an advantage by 
seme and a disadvantage by others. 

Weighting, whether simple or regression-based, is probably not the 
final word on performance monitoring. Even with much refined weighting 
schemes, seme problems with outcome indicators are likely to remain. 

First, it is not clear that weighting or regression adjustment en 
fully solve the problem of comparisons across areas and time. Even with 
regression adjustment, local performance scores can vary in ways that do 
not reflect real performance.^ Second, although the indicators tested 
often ranked fair or good, none of them reliably indicated whic 1 particular 
employed enrollees were the ones impacted by the program and which would 
have found jobs anyway. The weighting approaches tested can therefore 
serve only as rough guides to local operators in allocating program slots 
among enrollees. For low-cost components, encouraging broad targeting with 
weighted indicators may be adequate. But for high-cost components, it 
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becomes much more important to know who precisely will succeed by 
participating — and only by participating. For that purpose gross 
demographic characteristics alone may not provide sufficient guidance. 
Future research in this area may focus instead on more complex methods of 
making individual targeting decisions, comparing formal and informal 
techniques of assessment, and evaluating screening and filtering devices 
such as preliminary job search or other low-cost emoloyment-or iented 
activities. 

II. Participation and Coverage 

Performance measures based on program participation have >ften been 
user 3 as an alternative or compleme; to employment and welfare outcome 
measures. Compared to outcome measures, participation rates have both 
advantages and disadvantages. One clear advantage is that participation 
can be easily observed in 'cne short term. Management control over the 
day-to-^ay operation of a program can therefore be readily achieved by 
monitoring participation, in programs intended to be mandatory, monitoring 
participation may also b* undertaken to ensure compliance among enrollees 
and equitable treatment on the part of operational units. And biasing the 
program toward serving the most job ready is pi^bably not as strong with 
participation standards as with simple job entry or other outcome 
standards. 

The impact findings imply th^t administrators who track participation 
can achieve positive results by aiming for broad involvement over a wide 
range oi client types. At a minimum, participation standards should 
discourage concentration only on the most job ready. MDnitoring total 
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participation may not by itself be sufficient to accomplish this. 

Monitoring participation separately for key subgroups provides necessary 

information about what kinds of enrollees receive services and what kinds 

remain inactive. Participation standards can then be established to adjust 

the priority given to individuals with, for example, poor work records or 

long welfare history. Priorities for service can be set explicitly. Or a 

weighting scheme like that just applied to job entries can be used to 

provide incentives in a less rigid fashion. 

Seme important questions about standards for participation are not 

answered by this study. On the operational level, it has often been 

observed that the actual nature and intensity of participation for 

components with the same name varies widely across localities. For 

example, the nunber of employer contacts required of registrants in 

independent job search is small in some areas, large in others. For group 

job search, there is variability in the content of sessions and the amount 

of time spent in attendance. Quality of participation may therefore be 

important to monitor, but not much can be said about this topic here. 

The participation rates given in Chapter 2 nay not be a suitable guide 

for planning. Those rates were calculated for research purposes. They are 

"ever participated" rates rather than the point-in-time participation rates 

usually available to program operators. In addition, comparisons across 

localities are not straightforward, just with outcome indicators. 

Determining reasonable overall participation goals will hinge on 

characteristics of elig .bles, local labor market conditions, program 

objectives, and available resources. These problems are taken up in 

6 

greater depth in a forthcoming MDRC study. 
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Participation monitoring faces one important conceptual issue not 
faced by outcome monitoring. The absence of a job entry clearly indicates 
the absence of a favorable employment impact for an individual. In 
contrast, nonparticipation does not necessarily mean that a program has 
fa iled to ■ reach ■ an indiv idual as planned . in mandatory programs , 
sanctioning and other program contacts with nonparticipants are explicitly 
intended to handle noncompliance and to affect the behavior of enroilees. 
Some program impacts are therefore expected on nonparticipants. In addi- 
tion, enroilees may find work or leave welfare in lieu of participating, 
and these responses are also part of program impact. For these indi- 
viduals, the program objective is achieved without participation; having 
them participate would not only increase program costs but might also delay 
the employment or case closure outcome. Such contingencies are not 
accounted for under common operational definitions of "participation." 
Moreover, a drive for maximum participation may not be efficient. It may 
result in wasteful program expenditures on many enroilees who would have 
become employed or left welfare without participating. 

To handle the difficulty, KDRC has used the concept of program cover- 
a^e. Coverage measures have considerable potential, although to date they 
have been used only in evaluation research for the "DRC Work/Welfare demon- 
stration. These measures count, in addition to instances of participation 
£er_se, cases with some acceptable substitute for participation or where 
sanctions for nonparticipation have seen imposed. The concept coverage 
♦*akes into account normal welfare caseload turnover, but it does so without 
requiring information about prior employment and welfare and it need not 
involve weights. 
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Under a coverage formula, a client might be counted as 'covered* by 
program requirements if any of these outcomes is achieved; 

1. Participates in program activities 

2. Becomes employed 

3. Leaves AFDC 

4. Is sanctioned for nonparticipation 

The incentive effects of coverage standards are opposite to those of job 
entry or case closure standards. To maximize coverage, the attention of 
administrators is automatically directed to potential .longer-term 
recipients. Individuals remaining on welfare only a short time will 
automatically be counted as coverea when they leave AFDC; provision of 
serviw^s for them will not aid to the program's coverage score because they 
will already be counted. On the other hand, those individuals who have 
longer expected length of st.ay on welfare can be covered only if they are 
reached by some program component. A coverage standard therefore carries a 
built-in incentive to work with individuals below the top tier. 

The following example suggests how a coverage measure might work in 
practice. In the five urograms studied, only 5 to 25 percent of experi- 
mental were still on welfare nine months after enrollment and had not 
begun employment, had not participated in any major component, or had not 
been sanctioned for not participating. Thus, the 9-month coverage rate 
ranged from 75 to 95 percent of enrollees. These rates can convey a 
meaningful overall impression to legislators and the public about hew well 
a program is reaching its eligible caseload. In addition, the goal o£ 
increasing coverage would shift attention toward more dependent subgroups 
because they are typically on welfare and enrolled in the program longest. 
In this connection, it is noteworthy that two-thirds of the *not-covered" 
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experimental were recipients, and three-quarters of this group had no 
7 

prior earnings* 

There are, however, important disadvantages to the use of coverage 
indicators. Operationally, this class of measures presupposes the 
capability to follow the participation status of individual enrollees over 
time, which would require expenditures on setting up and operating tracking 
systems. Moreover, the cost of collecting, coordinating, and quality 
checking data on participation, welfare receipt status, and time frcm 
enrollment — all of which go into the coverage statistic — may 
substantially exceed the cost of maintaining simple participation counts* 
Since coverage rates are highly sensitive to normal welfare turnover rates 
and area labor market conditions, they would not solve the problem of 
comparisons across localities. 

In conclusion, two limitations of this study should be noted* First, 
no mention has been made of intermediate outcomes, such as literacy and 
basic and specialized skills. Monitoring improvements in these is iDecoming 
an increasing preoccupation of program managers. Seccnd, the nature of 
the programs included in the study prevented investigation of targeting for 
more expensive education and training services. 

This report does not put forward one ideal set of program performance 
standards. It recognizes strengths and weakness in alternative measures. 
At the same time, it has endeavored to evaluate sane general principles. 
The most fundamental is that in welfare employment programs performance 
measures should take account of differences in the job readiness and 
welfare dependency of the individuals served. They should do so in a 
manner that counteracts the "ccramonsense" notion that the best program. 
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results come from the top tier of eligibles. For this purpose, weighted 
outcome and participation measures correct sane defects in the incentive 
properties of common unweighted measures. Coverage measures also hold 
promise. 
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APPENDIX A 

STATISTICAL CONSIDERATIONS IN SUBGROUP COMPARISONS 



This appendix considers some special statistical issues which arise in 
the analysis of subgroup impacts, its purpose is to lay out the justifica- 
tion for the conclusions presented in the report summary. To do so, it 
focuses on the estimates for low-, mid- and high-dependency rankings shown 
in Table 4,9. Some tests of impact differences across selected subgroups 
are discussed. Approximate tests of statistical significance e^s examined 
to account for multiple impact estimates. Similar reasoning could be 
applied to any of the subgroup impacts presented elsewhere. 

When impact estimates are available for an entire population, statis- 
tical tests are unnecessary. Whatever the estimate is, whether it is large 
or small, it may be accepted as the true aggregate program effect with 
certainty, assuming that the estimate was produced in a valid fashion. But 
estimates based on samples, rather than on an entire population, contain an 
element of chance. Tc help rule out the chance element and increase confi- 
dence in any recommendation for service priority, certain statistical 
principles can be applied to impact estimates. Basically, the larger the 
estimated impact for a particular subgroup, and the more consistently it is 
found across samples, the more likely it is that the program model is gener- 
ally effective for that subgroup. in addition, the larger the estimated 
difference between the impact for a particular subgroup and the impact for 
the other subgroups, and the more consistently this difference is found 
across samples, the greater the confidence program planners may have that 
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granting priority in services to that subgroup will maximize the effect of 
the services in question. 

Two kinds of statistical .ests of impacts are relevant for this discus- 
sion. First, the basic experimental-control difference (the estimate of 
program impact) for a subgroup can be used to evaluate the hypothesis that 
the subgroup obtained no program effect. This is the usual t-test, applied 
to the magnitude of the difference between the estimated impact and zero 
impact. The same kind of t-test may also be applied to the difference 
between the estimated impact and any other fixed number, such as $100 for 
earnings. Second, the magnitude of two impacts for different subgroups can 
be compared. This kind of test is necessarily much less precise for any 
given sample because it involves a comparison of two estimates rather than 
a comparison of one estimate with a fixed number. 

Because there is more than one set of subgroups, an additional compli- 
cation arises. It is clear that if an unlimited number of subgroups can be 
examined, the'" sooner or later some will turn out to show a statistically 
significant effect, even if chance alone is operating. This problem occurs 
in all research involving multiple comparisons, and it requires that the 
usual statistical tests be qualified and made more stringent. 

The statistical criteria considered here are chosen to test whether 
certain targeting strategies are likely to achieve the results they are 
intended to achieve. There are several reasons why program planners* might 
adopt a strategy of targeting. One reason is that they wish to maximize 
program effect on employment or welfare receipt, given the limited 
resources available for services. This is the reason which is addressed in 
this report. Several alternative statistical criteria — some strict, some 
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loose — may be adopted to decide whether a particular group merits a recom- 
mendation for priority targeting in order to maximize program impact. Pass- 
ing any particular criterion means that the associated level of confiderce 
in the conclusions has been attained. 

Perhaps the strictest criterion would be to require that a subgroup 
shew consistently and statistically significantly greater impacts than the 
balance of the sample before stating that one particular subgroup has 
larger impacts than the rest. A less strict criterion would be to require 
the subgroup to show consistent and statistically significant impacts 
compared to zero — that is, that the experimental-control difference for 
that subgroup be generally statistically significant. A still less strict 
criterion would be to require the subgroup estimates to be larger than the 
balance of the sample, but without exceeding usual statistical standards. 
The least strict criterion would be for estimates to exceed zero but 
without passing any statistical test. 

It is the view of the author that only aubg^oups passing the most 
stringent criterion deserve a strong recommendation for priority targeting 
The consistency requirement is, however, quite difficult to pass even with 
samples of a thousand or more. Moreover, it is clear that when resources 
are limited, seme targeting decision must be made, and this necessity may 
legitimately warrant accepting a lower level of confidence until additional 
research findings become available. On the other hand, it is the author's 
view that if only the least strict criteria are passed, then recommenda- 
tions to target, even with resource constraints, will not be appropriate* 
It should be added, that even if a stringent criterion is passed, there may 
be other considerations which figure prominently in a targeting decision 
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and which qualify or override conclusions based on the impact data alone. 

The first targeting hypothesis, suggested by common practice, is that 
it is worthwhile to focus attention on the most employable and least depend- 
ent enrollees. Given the preceding discussion, this hypothesis should be 
accepted only if impact estimates for these subgroups either exceed those 
of other subgroups with some regularity cr exceed some cutoff value or, at 
a minimum, exceed zero by a statistically significant amount in a more or 
less consistent fashion. It is clear from inspection of Table 4.9 and 
other tables that this hypothesis is untenable. Not only do subgroups of 
the first tier fail to exhibit impacts above average or above the balance 
of the sample, but their estimates are also generally below average and are 
sometimes the lowest in any subgroup comparison. 

The second hypothesis, which derive from previous empirical analyses 
of the distribution of expected welfare tenure across welfare subgroups, is 
that the least job-ready or most dependent should be given priority for 
services. The estimates in this report do not strongly support this 
hypothesis. Earnings impacts for subgroups classified as relatively 
dependent were not the largest and did not generally exceed the 
mid-dependency subgroups. Welfare savings did appear relatively large, 
especially in comparison to earnings, for sane dependent subgroups, but 
this pattern was not consistent. Thus, although the observed savings are 
in line with theory, the estimates fail to pass either of the more 
stringent criteria established above. The inconsistency in welfare savings 
across sampies reduces confidence and increases the risk of offering 
low-cost programs primarily to one of the most dependent subgroups. 

Earnings impacts for subgroups in the middle dependency tier cane the 
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closest of the major subgroups to passing a <:est for priority services. 
Applicant returnees, in particular, show the roost consistent earnings 
impacts across samples and exceed the $100 cutoff in the four programs that 
had overall earnings *ains. Table A.l presents some statistical tests 
comparing impacts for second-tier subgroups with those for subgroups below 
and r-bove them. The table shows the sign of the impact differential 
between subgroups and its level of statistical significance and also gives 
the probability value associated with this cross-subgroup t-test. For 
example, applicant returnees had impacts greater than the remaining 
applicants plus all recipients (i.e., the balance of sample) in the three 
samples where such a comparison was possible, and the differential was 
statistically significant in two (i.e., in Baltimore and Arkansas but not 
in Virginia). Yet even this subgroup does not pass the most stringent 
criterion, The balance-of-sample test fails in Virginia, and the test of 
returnees against first-time applicants in Can Diego (the only possible 
test there) is not statistically significant. Moreover, the impact for 
this subgroup in Cook County would not likely be statistically significant- 
ly positive, even if the necessary information were available to identify 
the group there. 

A less stringent criterion is that impacts be consistently positive 
for a subgroup. Applicant returnees come closest to passing this criteri- 
on. In four of the five programs — and in all four programs with overall 
positive and statistically significant impacts — the estimates of earnings 
impacts are positive and statistically significant. Allowance should, 
however, be made for multiple comparisons in deciding whether this string 
of impacts is not the product of chance. There are, in fact, 14 major 
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TABLE A.1 

AFDC APPLICANTS ANO RECIPIENTS: TESTS OF DIFFERENCES IN IMPACTS 
ON EARNINGS AND AFDC PAYMENTS BETWEEN SELECTED MAJOR SUBGROUPS 



Subgroup Comparison 



Appl leant Returnees 
VERSUS 

All Other Subgroups 



Other ApolicantB 



Test of Difference Between Querterly Eernings Impacts 



Sen Diego Beltimore Virginie Arkansas Cook County 



+ 

1 .27] 



+ * 

[•08] 

+ 

(.64] 



+ 

(.44] 
+ 

(.32] 



(.02] 
+ 

(.04] 



Recipients 




( .06] 


+ 

(.54) 


(.03) 




AddLi* cants w4t fc i Lrkr t-hon 
$3000 Prior Yeer Eernings 
VERSUS 

All Other Subgroups 




+ • 
(.01] 


( .10] 


+ 

{ .14] 


- 99 

(.02) 


Other Applicents 


+ * 

f .06 1 


+ ♦* 

f n ^ 1 


- 99 9 

t n n 1 


+ 

f /in l 

I .40 J 


( .15] 


Recipients 




+ 49 

(.01] 


(.29] 


+ 

M7] 


~ 99 

(.02] 


Applicant Returnees with 
Less then $3000 Prior 
Yeer Earnings 
VERSUS 

All Other Subgroups 




+ ** 
(.02] 


(.43] 


+ *• 

( .03] 




Other Applicents 


+ 

■ .53] 


+ 

1.15] 


( .23] 


+ * 
( .06] 




Recipients 




+ *• 

(.02] 


1.61] 


+ 

( .04) 





(conti nued ] 
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TABLE A.1 (continued ) 





Test of Difference Between Querterly AFDC 


Peyments Impects 


Subgroup Comparison 


Sen Di ego Beltimore 


V i r g i n 1 e 


Arke nse s 


Cook County 


Applicant Returnees 
VERSUS 

All Other Subgroups 


... 

( .41 ) 


( .87) 


+ 

( .13) 






Other Applicants 




+ 

f DO 1 


+ 

:.46j 




_ 


Red pi ents 


( .35) 


( .90] 


+ • 

( .P6 ) 







Applicants with Less then 
$3000 Prior Year Earnings 
VERSUS 

All Other Subgroups 


( .20) 


( .53) 


i 

(.21) 




( .52) 


Other Applicents 


[ . 35 ] f .41 1 


f .32 1 


I .59) 




f oil 

I .31 J 


Red pi ents 


(.22) 


(.67) 


+ 

(.11) 




(.62) 


Applicant Returnees Kith 
Less then $3000 Prior 
Yeer Earnings 
VERSUS 

All Other Subgroups 


... 

( .41 ) 


( .59) 


+ 

(.20) 






Oth er Ap pM c e nt s 


(.21) (.78) 


( .49) 


+ 

( .59) 







Red pi en t s 


(.32) 


( .68) 


+ * 
( .09) 







SOURCE: 6oe Table 4*1. 



NOTES: The table shows the signs end statistical significance of the Impect of 
the subgroup at top of each penel minub the Inpect for the subgroup below it. Querterly 
Impacts are an average of querters four through lest. Two-teUed t-tests were performed 
for eBch program sample from e regression on pooled eppUcant end recipient dete, except 
In San D1ago f where recipients were not enrolled. Nuaerlcel values 1n the parentheses 
ere the probabl 1 1 11 as associated with the t-velues. Stet1ot1cal significance levels are 
Indicated as: * = 10 percent} ** = 5 percent; *** = 1 percent. 
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subgroups and subgroup combinations considered: 5 prior earnings categor- 
ies. 5 prior welfare categories, and 4 combinations of prior earnings and 
prior welfare (referring to Tables 4.2, 4.3 and 4,4). These groups are not 
independent, but assuming that they are permits a conservative, if 
approximate, accounting for the multiple canpariscns. 

The odds of finding four out of five sample estimates greater than 
zero at the observed levels are quite low, even with 14 independent 
trials. 1 Consistency of impacts for this subgroup across program samples 
is less certain. For one thing, the weak earnings results for Cook County 
applicants make it virtually certain that at least one of the five programs 
failed to achieve earnings gains for returnees. Disregarding the estimates 
for ' ok County, which did not obtain earnings gains for the sample as a 
whole, the likelihood that one of the other four programs obtained no real 
earnings effect for returnees is (LOSS,*" which parses the conventional 
statistical test criterion of 0.100. 

This test is by no means satisfactory, however, since it indicates 

only that working with this subgroup should produce earnings impacts, not 

that these? impacts are likely to be relatively large. A more stringent 

test would be to apply the $100 cutoff to subgroups in the four programs 

that achieved overall earnings gains, still assuming 14 comparisons. The 

probability of finding the observed array of earnings impacts for applicant 

returnees in the first four programs under the assumption that impacts for 

3 

all subgroups are $100 is under 1 percent. A more difficult test to pass 
is created by averaging the earnings impacts across the four samples. Even 
accounting for multiple comparisons, the resulting average exceeds $80 at 
conventional I >vels of statistical significance, although not the higher 
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$100 cutoff. Finally, the odds that impacts were below these cutoffs in 
none of the four samples are on the order of 50-50, implying that even this 
subgroup does not pass the most stringent test of consistency. 

This analysis of the principal results from a purely statistical point 
of view leads to the conclusions stated in the report. They imply that any 
strategy of priority targeting focused on the most employable or least 
dependent is not supported by the data, whatever the statistical criterion. 
The same applies to strategies to exclusively target the least employable 
and most dependent. Furthermore, when it canes to the most stringent 
criteria, none of tne subgroups examined emerges with certainty as the best 
targeting choice. 

Nevertheless, with lower degrees of confidence the applicant returnee 
and other mid-dependency groups may be identified as suitable for priority 
in services when resource constraints require that a choice be Fade. Weigh- 
ing the constellation of statistical evidence together with other considera- 
tions, this report concludes that a recommendation for exclusive targeting 
is not supported; that there is no certain best choice for exclusive target- 
ing; but that when targeting is imposed by a scarcity of resources ne mid- 
dependency subgroups are a suitable starting point for seeking earnings 
gains with the kinds of low-to-moderate cost programs under study. For 
welfare savings the evidence supporting any exclusive targeting scheme is 
weak. The more dependent subgroups are likely to assume increased 
importance in achieving welfare savings, although the results were, again, 
inconsistent across samples and suggest some risk in working with the third 
tier. 
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CHAPTER 1 



1. Friedlander and Long, 1987. 

2. Results of additional benefit-cost analyses were carried out 
by MDRC, which are quoted as relevant to this report. 

3. The use of the term "placement" is avoided in the rest of this 
report. The term was originally used by the employment 
service to denote referral of a client to a particular job 
opening by program staff. It is therefore inappropriate for 
programs that rely on a client's own job search efforts. In 
addition, placements, or self-reported employment, tend to 
understate employment and earnings because recipients sane- 
times do not report jobs to welfare staff, or leave the 
program before they find a job and hence are not obligated to 
report their employment. 

Similarly, the term "of f-welfare" is used rather than "case 
closure" because it is mere inclusive. It covers persons who 
apply for AFDC, enter a program, but then quickly leave the 
welfare system without having been approved for a grant (i.e., 
without ever having had a case opened). 

"Off-welfare" is not identical to the "welfare reduction" indi- 
cators in use. The former looks only at whether families are 
receiving any AFDC payment, and can be stated either as a 
numerical connt or as a percentage. The various welfare 
reduction formulae in use subtract pre-program welfare grant 
levels for clients from their post-program welfare receipt to 
arrive at a dollar figure, either aggregate or per registrant. 
This study tested an off-welfare indicator rather than a 
welfare dollar reduction indicator because tne pre-prograra 
data necessary to simulate the latter were lackir^ from the 
San Diego and Baltimore research data bases. 

4. see Bane and Ellwood, 1983; Ellwood, 1986. 

5. See Ellwood, 1986, p. xii. 

6. See O'Neill et al., 1984, p. 84. 

7. The role of performance scores in the actual distribution of 
funds has been quice small. The bulk of federal Will funds 
have been allocated to states according to number of Witt 
registrants. On the basis of budget appropriations during t**3 
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1970s, it has been determined that incentive rewards for 
performance based on this formula could amount to about 
one-third of all federal WIN moneys given to states. (See 
Office of Family Assistance, 1985, pp. 13-14.) 

In practice, annual funding changes have been restricted in 
other ways. WIN regional coordinators have had discretionary 
powers, and incentive moneys could be allocated for local 
performance achievements not incorporated in the mathematical 
formula or on the basis of other considerations. As a result, 
only about 3 percent of funds distributed in a given year have 
reflected performance scores, although cumulative changes 
across the years could have amounted to more. (Office of 
Family Assistance, 1985, p. 21.) 

Job retention has been a more important determinant of the 
program performance score in the discretionary part of the WIN 
Allocation Formula ';han job entry, although there is some 
evidence that the complexity of the formula kept this fact 
hidden f ran line operators (Mitchell, Chadwin, Nightingale, 
1980, p. 287). The relative potential of each element of the 
formula to raise a state's overall performance score differed, 
depending on how high or low its score on each eisment might 
be. The complexity of the discretionary part of the formula 
was such that determining which elements had the greatest 
influence on scores would be very difficult without 
sophisticated analysis and simulation. 

Participation is observed now, whereas outcomes may be 
observed only after sane months and may require suhstantial 
effort in locating clients to ask about their employment 
status. Monitoring subgroup participation may be the most 
effective way of ensuring local compliance with an optimal 
targeting plan. 

The problem of specifying optimal performance standards for 
independent local service providers for JTPA programs has been 
highlighted by the growing use of fixed-priced contracting. 
The language of JTPA has encouraged the use of fixed-priced 
contracting because all costs i: cur red can be allocated to 
"training, ■ thus helping programs to comply with the 15 
percent cap on administration costs. For a thorough 
discussion of the possibilities and problems in fixed-priced 
contracting see Wallace, 1985. 



Goldman, et al., ?986, p. 92. Indicators that make use of 
pre-program client measurcc arc often referred to as change- 
based indicators, with simple outcomes designated as level 
indicators. The example given in the text of this chapter for 
San Diego would suggest that change-based indicators should 
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prove superior to simple outcomes as proxies for real program 
impact. In that case, the change frcm no pre-program employ- 
ment to employment during the follow-up period was associated 
with the larger program-induced impact on employment. The 
weighted job entry rates tested in this paper are change-based 
indicators, since they award more performance points for the 
employment of clients who were not employed in the recent 
pre-program period. 

The relevant literature on indicator validation is based on 
several analyses of CETA. Borus, 1978, found that job entry 
had very little pcwer to indicate net impact for CETA. Gay 
and Borus, 1980, in a study of four pre-CETA programs, found 
change indicators to be somewhat superior, and rated simple 
job entry as one of the poorest measures. In contrast, Geraci 
and King, 1981, found evidence supporting job entry as the 
better measure, as did Geraci, 1984. Zornitsky efc al., 1985, 
produced results favoring level indicators. The latter three 
studies also concluded that post-program follow-up added 
valuable information about employment at the point of 
termination. 

These studies all suffer sericus methodological problems from 
havi' 5 been based on non- experimental impact estimates. The 
prin .ipal issue — the Value of level indicators versus 
change-based indicators — is still the most pressing one to 
be resolved in performance monitoring. The issue is compli- 
cated by the possibility that the best class of indicators may 
be different for welfare women, adult men and youth. Adult 
men entering employment programs typically exhibit a temporary 
pre-program dip in earnings, making prior earnings problematic 
as a proxy for earnings capability. Youth often have short 
and erratic earnings histories, and a pre-program earnings 
baseline may therefore be meaningless fo,; them. 

11. See Gc^an, 1981; Wolfhagen, 1983; MDRC, 1980. 



CHAPTER 2 

1. See Goldman et al., 1986; Friedlander et al«, 1985b; Riccio et 
al. , 1986 ; Friedlander et al. , 1987; Friedlander et al. , 
1985a. (x ? or a summary of the demonstration's findings thus 
far, see Gueron, 1987). 

2. In this report, participation and sanctioning rates were 
calculated on .somewhat different bases than in the published 
state reports. In this study, the base is always "all 
experimental. " In the state reports, the base of "all 
program registrants' was often used. Most experimental did, 
however, register for the programs, and the differences 
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between the figures cited h*re and those published in the 
state reports are not large. 

3. In San Diego, a second experimental group received job search 
only. The program and its evaluation were also carried out 
for AF" -Us. Neither of these research groups is analyzed in 
this study * 

4. Cook County, like San Diego, had two research groups, one to 
test job search plus work experience and the other to test job 
search alone. Only the former is included in this subgroup 
study. The Cook County WIN Demonstration program also worked 
with AFDC-Us, but this part of the caseload was excluded from 
the evaluation. 

5. Sample sizes in this report differ slightly from those in the 
corresponding state reports. An attempt was made here to 
assign values to demographic data where these were missing. If 
missing data could not be inferred with reasonable certainty, 
the cases were dropped from the analysis. The effect of this 
strategy on sample size was the gain of 7 cases in San Diego, 
54 cases in Baltimore, and 8 in Arkansas, but a loss of 32 
caser in Virginia. The Chicago sample was unchanged. 

Randomization produced similar experimental and control groups 
with some differences. There were small differences between 
research groups in ethnicity and marital status in the San 
Diego sample. In Baltimore and Virginia, small differences 
were apparent in measures of education, prior employment and 
earnings. 

6. This does not mean that the indicated subgroups account for 
the bulk of all AFDC expenditures. Benefits paid to families 
outside the WIN-mandatory sample are not counted. Nationally, 
about two-thirds of AFDC families are WIN-exempt. 

7. In discussing the ability to predict differences in behavior, 
a distinction must be made between individuals and groups. As 
shown in the text, a wide range of differences in average 
outcomes across groups can be predicted quite well. 
Differences among individuals are less predictable. 

8. Prior work ar*d welfare histories are important in these adult 
samples, but may not play the sa;ne roles for a sample of 
younger mothers. Youth, simply because they are young, often 
have short work and welfare histories making these predictors 
less powerful than for adults. 

9. Younger children are defined here as 12 years or under. 
Since, with the exception of Arkansas, these programs employed 
the traditional WIN definition of mandatory, women with 
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children under age six were largely exempt. Those few who are 
in the study samples are probably not representative of the 
rest. Restriction of the study sample to WIN-mandatories 
implies that correlations of demographic characteristics with 
tuture employment and welfare receipt • not be represent- 
ative of the AFDC population at la* ^e, or of the wider 
population of poor family heads. 

10. An inference based on the observed patterns of participation. 



CHAPTER 3 

1 . For more complete reports of data quality control, see the 
individual state reports. 

2. For more detail about data sources and follow-up, consult the 
state reports. 

3. The distinction between unconditional and conditional impact 
estimates can be develop 3 as follows. The basic impact 
regression model is 

Y(T, SI, S2, X) 

where 

Y outcome variable 

T experimental group dunmy variable 

51 dummy variable for subgroup dimension 1 

52 dummy variable for subgroup dimension 2 

X vector of additonal control variables 

The full sample impact is the co fficient of T. The uncondi- 
tional subgroup estimates for SI come from the regression 
model 

Y(TS1, TNS1, SI, S2, X) 

where 

TSJ = T * SI 

TNS1 = T * (1-Sl ) 

The impact on groups Sl=l and S1=0 are read from the 
coefficients of TS1 and TNS1, respectively. Finally, the 
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conditional rriodel i - 

Y(T, TS1, TS2, SI, S2, X) 



where 

TS2 = T * S2 

and the coefficient of T is the impact when S1=0 and S2=0. 

The coefficient of TS1 is the additional impact attributable 

to the SI characterstic when S2 is held constant. The 
coefficient of TS2 is the additional impact attributable to 

the S2 characteristic when SI is held constant. 

Interactive specifications are possible for both unconditional 
and conditional models. For the unconditional case, 



where 



Y(TS12, 


TS1N2, 


TSN12, TSN1N2, 


TS12 


= T * 


SI * S2 


VS1K2 


= T * 


SI * (1-S2) 


TSK12 


= T * 


(1-S1) * S2 


TSN1N2 


= T * 


(1-S1) * (1-S2) 


S12 


= SI * 


S2 



For the conditional case, 

Y(T, TS1, TS2, TS12, SI, S2, S12, X) 

Coefficients in this latter model can be combined to reproduce 
the unconditional interaction estimates exactly. But when a 
third subgroup dimension is introduced, S3, the terir. TS3 in 
the conditional model would make the two sets of interaction 
estimates different. 

4. See Borus, 1978. 

5. Individual impact estimates are made by (1) regressing demo- 
graphic and background characteristics on employment and 
welfare outcomes for the experimental and control groups , and 
then (2) using the coefficients obtained f ran these regres- 
sions, along with the characteristics of individual members of 
the experimental group, to predict individual impacts. The 
first stage estimate is made from the conditional subgroup 
impact regression model. That is, fran the regression that 
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contains the full array of experimental subgroup interactions, 
a prediction is made for the expected program impact on earn- 
ings and welfare receipt for each person in the experimental 
sample. The net impact estimate will differ for each person, 
depending on the demographic and prior work and welfare 
characteristics at the time of entry into the research sample. 

These are sometimes referred to as direct estimates . For 
example, with treatment interactions for prior employment, 
education and number of children, one impact would be 
predicted for an experimental with no prior employment, no 
diploma, one chil^; a different net impact would be predicted 
for an experimental with any difference in any of these 
characteristics. The more variance in the dependent variable 
that can be accounted for by the regression model, the better 
the predicted net impacts* At the present state of knowledge, 
however, most of the variation in the outcome measures cannot 
be explained. 



1. For this analysis, earnings impact regressions were run on the 
pooled sample of applicants and recipients, separately in 
Baltimore, Virginia, Arkansas and Cook County. The model 
specified an experimental group durrmy, a dummy for applicants, 
and a dummy for an experimental-applicant interaction. This 
last dummy gave the estimate of the unconditional impact 
differences. Interactions of experimental group membership 
with alJ other subgroup characteristics were then added and 
the same coefficient read again. The coefficient changed very 
little. The t-statistic for this coefficient gives the statis- 
tical significance of the conditional difference in impacts 
between applicants and recipients. Applicant/recipient differ- 
ences in earnings gains were statistically signif icant in 
Baltimore and Cook County but not in Virginia and Arkansas. 

2. This analysis of subgroup combinations intentionally does not 
break up the first-time applicant group into prior earnings 
subgroups. Continually subdividing the samples in this 
fashion quickly reduces sample sizes below the point where 
statistical analysis is meaningful. Moreover, it ic question- 
able whether very small subgroups can have much policy 
importance for mass participation programs. 

3. Friedlander, et al., 1985b. 

4. Friedlander, 1987. 

5. AFDC benefit levels also varied across counties in Virginia. 
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6. See Riccio et al., 1986, p. xiv. 

7. Length of employment usually means job retention, i.e., remain- 
ing continuously employed with a particular employer. it is 
difficult to identify this kind of job retention with UI earn- 
ings data. It is, however, possible to examine other measures 
of ^achment to employment. For example, it was found that 
30.3 percent of applicant experimental worked in six or more 
quarters from quarter K through the last follow-up quarter, a 
statistically significant increase of 8.5 percentage points 
over applicant controls. The corresponding level for recip- 
ient experimentals was 11.7 percent, and this represented an 
increase of only 2.7 percentage points over rv^cipient controls 
(not statistically significant). 

8. To determine the relationship between impact on employment and 
impact on welfare receipt for recipients, short- and longer- 
term employment variables were added to the right-hand side of 
the welfare impact regression. The resulting equation may be 
interpreted as part of a recursive model of program effects: 

E(T, X) 
A(E, T, X) 

where the variables are defined as follows: 

E = the set of follow-up employment and earnings 
variables 

A = the set of follow-up welfare receipt 
variables 

T = the treatment dummy 

X = the set of control variables. 

As elsewhere in this report, A *as defined as welfare payments 
for quarter 4 through the last follow-up quarter. Several 
specifications for E were tested to determine how their 
introduction into the second equation would change the 
coefficient of T from the simple impact equation A(T, X). The 
largest change was to reduce welfare savings for Arkansas 
recipients from $60 per quarter down to $46, a decline of less 
*;han 25 percent. Thus, at least three-quarters of the impact 
on AFDC payments for this subgroup came through progrer 
effects other than an increase in UI earnings. 

9. See Friedlander et al., 1986. 

10. For the method of calculation of this percentage, see note 15. 
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In San Diego, the total dollar savings for unnmarried women 
were 123 percent of the total for the full sample. For 
Virginia, this figure was 104 percent, and for Arkansas, 88 
percent. The unweighted Average of these three is 105 
percent. This figure may be interpreted to mean that if these 
programs had served unmarried women in the same way they did, 
but served no one else, then their total welfare savings would 
have been slightly higher than they were. This assumes no 
interactions between subgroups. For never-married women, the 
shares were 39 percent for San Diego, 87 percent for Virginia, 
and 91 percent for Arkansas, giving a simple average of 72 
percent. 

11. This test of statistical significance is not relevant to the 
hypothesis that impacts are smaller for women with a 
pr e-school child. There is a high degree of confidence that 
this hypothesis is not true for this sample simply because the 
employment and earnings effects are larger for this subgroup 
rather than smaller. The absence of statistical significance 
means only that it is not certain whether the impact for this 
subgroup would tend usually to be larger on repeated trials. 

12. It was, moreover, not possible with the available data to 
identify any subgroup of applicant returnees, or certain of 
the other subgroups examined above. Readers of the final Cook 
County report (Friedlander et al., 1987) will recognize that 
"applicants" were then labeled "new recipients" and 
"recipients" were called "prior recipients." 

13. Friedlander et al., 1987, pp. 80-83. 

14. The odds of one or more of the subgroups attaining earnings 
gains statistically significant at the level indicated purely 
by chance are less than one in ten. 

15. The total earnings impacts for a program were calculated by 
multiplying the average impact per experimental by the number 
of experimentals. The portion of the impact associated with 
applicant returnees was calculated by multiplying the 
per-experimental impact for this subgroup by the nunber of 
experimentals in the subgroup. Division of the latter amount 
by the fcrmer converts the share from a dollar figure into a 
proportion. These proportions were 89 percent for San Diego, 
69 percent for Baltimore, 49 percent for Virginia, and 77 
percent for Arkansas. The simple average of these is 71 
percent, the top end of the range cii ed in the tex^ 
Excluding San Diego, which did not work with recipients, the 
average is 65 percent, the bottciu end of the range cited. 
These figures should be interpreted to mean that if the 
programs had served applicant returnees exactly as they did 
and did not enroll any other subgroups, the total earnings 
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impact would be 65-70 percent as large as what was actually 
measured, assuming that effects for any subgroup are 
independent of effects for the others. 



CHAPTER 5 

1. Employment and welfare receipt rates were calculated for 
nonparticipants and participants in San Diego, Batlimore, and 
Virginia. No less than half of all job entries in each of 
these programs were obtained by experimental sample members 
who never became active in any formal program component. 
Nonparticipants outnumbered participants, and the percent of 
nonparticipants who held Ul-recofded employment at quarter 6 
was at worst only a few percentage points lower * for 
participants. In addition, nonparticipants were mor.. *.tkley 
to be off AFDC by that time. 

2. Under-reporting of job entries can occur when case heads who 
leave welfare because they have found jobs <to not report 
employment. Particularly in large urban areas with large 
caseloads, cases are often closed because the client fails to 
respond to seme attempt at contact, making it impossible to 
record employment status or other eligibility factors. In 
addition, reports of employment obtained by income maintenance 
staff for the purpose of adjusting grant payments are not 
always reported back to the staff of the employment program. 

3. Regressions for average earnings and average welfare payments 
over quarter 4 through the last quarter were run with all 
treatment-subgroup interactions in the model at ence. The 
coefficients of these interactions were then used to predict 
for every experimental group member the expected net impact on 
earnings and welfare receipt. These new variables were thei, 
correlated with employment and off-welfare status, using only 
the experimental group sample. 

4. These weights represent approximately the relationship of 
control group mean earnings for prior-earnings categories in 
the composite impact table in the impact chapter of the 
earlier report in this study. 

5. Sane of the variety in local job-entry rates cannot be 
accounted for by current regression models, and this problem 
becomes more severe the greater the degree of disaggregation. 
To examine this issue, the local office designators available 
in San Diego and Cook County were used. Regression-adjusted 
job-entry rates and average earnings impacts wera calculated 
for each office, using the subgroup demographics as regression 
control variables. A crude estimate of "mistakes" made by 
regression-adjusted scores may be made by counting the number 
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of office** with high adjusted job-entry rates and low impact, 
or vice ver' . In San Diego, there were 7 offices. Of the 3 
offices wit:* adjusted job-entry rates above the median, 2 had 
impacts below the median. Two of the 3 offices with job 
entries belcw the median had impacts above it. In Cook 
County, there were 11 offices. Two of the 5 offices with job 
entr^ rates above the median had impacts belc: the median, and 
1 office with job entries belcw the median had impacts above 
it. These results should be considered suggestive rather than 
definitive. Future research may well succeed in identifying 
labor market variables suitable for the local office level of 
aggregation; this analysis did not utilize such variables. 

6. Hamilton, 1988. 

7. Based on statistics for San Diego, Baltimore, and Virginia. 
The data for Arkansas and Cook County were not available in a 
form that permitted estimates to be s.iade. 

8. Intermediate outcomes are program objectives intended to lead 
eventually to employ Ment and departure from welfare. One 
major disadvantage in emphasizing intermediate outcomes, 
however, is that they do not necessarily bring about impacts 
on employment and welfare receipt. On the other hand, 
deficits in skills are more readily measured than deficits in 
" employ abil i ty. ■ Likewise, compared to the ultimate employ- 
ment and welfare outcomes, improvements in these kinds of 
intermediate outcomes are less likely to occur without some 
special training. For example, fluctuations in local 
unemployment rates or changes in family circumstance will 
greatly affect the probability of a job entry, independent of 
anything a program might do. But such fluctuations have 
little influence on reading level, which is likely to change 
only \*ith participation in some remedial activity. Thus, if a 
program obtains increases in reading level for a high 
percentage of enrollees, it ma* take credit for success. In 
theory, administrators may be more confident that this kind of 
success is a product of the program's efforts than they may be 
for job entry and off-welfare outcomes. 

In practice, this statement probably requires qualif ication. 
Enrollees even in mandatory welfare employment programs show a 
surprisingly high degree of s<flf -initiated educational 
activity. For example, control-group members in Virginia, 
where data was obtained on non-program educational activity, 
showed rates of activity as high as experimentals, who could 
be referred to educational institutions as part of their 
program participation. Similar data were available for Cook 
County as well, and educational participation reached about 17 
percent for experimentals and controls, even though the 
program itself did not actively refer clients. To the degree 
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that a program provides educational services tc individual 
who would have found them anyway, the improvement in skills is 
not a program effect. 



APPENDIX A 

This probability is calculated in the following fashion. 
First, the right-tail probabilities associated with the 
t-statistics for applicant returnee earnings .Impacts for San 
Diego, Baltimore, Virginia and Arkansas are multiplied by each 
other. These probability values are, respectively, 0,0058, 
0.0039, 0.0427 .nd 0.0032. For Cook County, the probability 
was assumed to be unity in the absence of actual data. The 
product is the probability of obtaining the observed impact 
values or higher, assuning that all the true values ar_- zero. 
The probability of obtaining lower estimates in 14 ind Pendent 
trials is the 14th power of 1 minus this probability. This 
result is then subtracted from 1 to yield the probability 
referred to in the text: less than one in a million. 

The likelihood that one of the four programs achieved zero or 
lower earnings impact for applicant returnees is 1 minus the 
likelihood that impacts were greater than zero in all four. 
The computation therefore begins by subtracting each of the 
four probabilities cited an the previous footnote from unity. 
The figure in the text is then 1 minus the product of these 
four numbers. 

The figure in the text is calculated as in footnote 1, substi- 
tuting the probability values associated with a t-test against 
$100 instead of zero. The four probability values are 0.1754 
for San Diego, 0.1063 for Baltimore, 0.4147 for Virginia and 
0.0744 for Arkansas. The result is 0.008. 

Using equal weights for each sample, the average exceeds $100 
at the 5 pert, rc level using a standard one-tailed t-test. 
The same resuK holds if inverse standard errors of the impact 
estimates are used for weights in order to minimize the 
variance of the average. Allowing for 14 repetitions and 
maintaining an experiment-wise error rate cl ID percent with a 
one-tailed test, t*iese averages are not statistically signi- 
ficant against the $100 cutoff, although they are against an 
$80 cutoff. This test is an extremely conservative approach 
to multiple comparisons and indicates an acceptable degree of 
confidence that earnings impacts for applicant returnees on 
average fall ibove at least the lower cutoff. 
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